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TECHNICAL MEMORANDUM 


PROGRAM RISK ANALYSIS HANDBOOK 
I. INTRODUCTION AND OBJECTIVES 


The article “GRO Project Beset by Complications” appeared in the June 14, 1987, 

Huntsville Times. The NASA-GSFC Project Manager, Jeremiah Madden, stated “the sheer magni- 
tude and complexity of the GRO program overwhelmed managers and engineers and obscured some 
of the program’s finer details.” These complications occur on all space and defense projects, 
especially those that use unproven technology, attempt a new mission, and/or scale up (or down) 
the size of the craft used — for example, the C-5A, the Trident Submarine, the Space Telescope, 
and the Stealth Bomber. The specific problems encountered on the GRO project are typical: 

• Manufacturing processes not well-understood. 

• Manufacturing problems due to materials faults/availability. 

• Lack of trained manufacturing work force. 

• Unexpected electromechanical interference between instruments, once integrated. 

• Redesign of components and tooling. 

These problems led to a modest cost growth of $380 million to $500 million, and a schedule slip 
from May 1988 to early 1990. Space Telescope cost growth is $500 million to $1.4 billion. The 
average cost growth observed in both NASA and DoD projects, from the start of Full-Scale 
Development (FSD) to the completion of prototype production, has been in the range 30 to 40 
percent. 

This type of track-record for Federal acquisition of large-scale systems has led to 
@>ngressional skepticism, public outrage, and occasionally loss of support for the continuation of 
the project. Since the 1960s, DoD and NASA project managers have sought out management tech- 
niques that will help them control cost growth and schedule slippages. Major General John R. 
Guthrie stated at the 1970 DoD Project Manager’s Conference that: 

“The most rudimentary sort of good risk analysis might have enabled us to avoid 

most of the pitfalls we have encountered. By rudimentary I mean — did we identify 

those items which were new and identify the impact on overall system performance 

if that particular component or subsystem were to experience difficulty?” 

Program risk analysis is an iterative process (Fig. 1) for identifying, quantifying, and 
managing the uncertainties associated with complex design and development programs typical to 




Figure 1. Risk analysis iterative process. 

NASA. Others have defined risk analysis as a systems analysis approach to risk, or as a collection 
of techniques to identify, quantify, and manage risk. In contrast to techniques for quantifying 
operational risk (e.g., failure modes and effects analysis, fault tree analysis, reliability analysis), 
program risk analysis deals with program cost, schedule, and technical performance estimates. The 
key question to be answered by risk analysis is what are the distributions of probability on the 
mature (achieved) values of each of these three random variables. In some risk analyses, two of 
these variables (say, performance and schedule) are thought of as fixed. Then the probability dis- 
tribution on the mature value of the other (cost) is derived. However, in a comprehensive risk 
analysis, the dependence among these variables is assessed and uncertainty in one affects the 
assessment of risk in the other two. 

NASA NMI 7100. 14A (Major System Acquisition) calls out risk evaluation as a criteria 
second only to performance for initiating FSD. MMI 7110.1 requires risk analysis and cost risk 
assessment for both formal project predevelopment reviews. Program risk analysis is useful to 
program managers as both a source of information on the program and as a decision-aiding tool. 
The reason is that it is a formal, systematic, and documented approach to dealing with uncertainty, 
versus “seat-of-the-pants” dealing with problems as they arise. It can be used in both Phases A and 
B, as requirements and design configurations evolve, for the purpose of early identification and 
resolution of technical uncertainties. Classic risk resolution strategies are parallel development, 
design/operations trade-offs, and development of back-up solutions. It is probably of most use for 
assessing the potential for cost and schedule slippages in programs moving into Phase C/D, because 
this is the point at which significant resource commitments will be made. Risk analysis can be app- 
lied to subsystems of instruments, individual technologies, manufacturing processes, and other el- 
ements that make up a program; however, its true value is in synthesizing the multiple uncertainties 
which arise when new missions are planned, and technologies must be integrated into a new en- 
gineering design. 
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Each task team leader and program manager (PM) at NASA/MSFC should become aware of 
the valuable information available from a program risk analysis. The risk analysis process is itera- 
tive and risk analysis should not be viewed as a one-time, check-the-box type activity. This 
handbook is written with the view of the PM as the consumer for risk analysis, and a range of 
options is provided so that the appropriate type of analysis, at the right level of detail, may be 
requested. 

This handbook is prepared as a guide and reference source for any NASA employee who is 
requested to perform a risk analysis in support of the PM. This individual will be referred to as 
“the risk analyst,” although he/she may be a cost analyst, schedule "analyst, program analyst, 
engineer, or scientist. An entire range of risk analysis tools will be provided, along with some 
guidance for selecting the appropriate technique for a given situation. However, it is always neces- 
sary that the risk analyst apply his judgment when initiating a risk analysis at the request of a PM. 
For example, he must decide what technique is appropriate, given the time available and the 
software tools he has at his disposal. Another key question is how much access to and cooperation 
from program personnel the analyst can expect; no meaningful risk analysis can be generating 
without repeated probing discussions with practically all program personnel in order to identify 
technical uncertainties and their potential cost and schedule impacts. Good relations between risk 
analyst and technical team members is essential if a valid, useful risk analysis is to be conducted. 
The only alternative, and one that works well, is for the risk analyst to be part of an ad-hoc team 
of experts, independent of the project, whose job it is to review the project and report to higher 
authority on its findings. 

The selection and support of a risk analyst (or risk analysis group) is an important step for 
large industry/govemment design organizations and laboratories. The risk analyst is the alter-ego of 
top management in his evaluation of a project, much as the quality assurance analyst is for a 
production facility. He is often feared, avoided, circumvented, and detoured. Other engineers 
generally view the risk analyst as a nuisance who will take their valuable time, produce nothing 
new, and perhaps misrepresent their professional judgment. Although it is desirable for the risk 
analyst to be involved early in programs, and have an established relationship with both PM and 
project personnel, the fact is that risk analysis is usually a last-minute effort, performed on an 
ad-hoc basis, prior to some major decision/presentation. The risk analyst is often not at all familiar 
with the technology or program being analyzed. He must therefore: 

1. Educate himself quickly. 

2. Acquire the data. 

3. Use the data in some pre-developed model. 

4. Present recommendations based on model output. 

The above discussion clearly reveals that conducting a risk analysis is not an easy job. The 
risk analyst must be an individual of highest quality in education, technical/program experience, 
human relations, and recognition of management needs. Many of these positions are filled by 
individuals with graduate-level training in statistics, operations research, or systems analysis. An 
undergraduate degree in engineering or hard sciences helps, but is not necessary. Individuals who 
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are enthralled with math models and/or computer techniques generally are not good risk analysts. 
The interaction with project personnel, the collection and verification of data, and preparation of 
finding into a format useful for management are the key activities of the risk analyst. 


II. TERMINOLOGY 


Several general references on program risk analysis are given in the Bibliography. The 
definitions presented in the following are stated in broad terms and would be accepted by anyone 
working in the program risk analysis area. Of course, certain organizations and programs within 
organizations adopt more specific definitions for terms such as “risk area” — in fact one of the 
first jobs for a risk analyst newly assigned to a program is to work with the PM on an agreeable 
set of definitions and working groundrules. Note also that more specialized terminology is used in 
health and environmental risk analysis, and these terms are not appropriate for program risk 
analysis. 

Definitions 

Risk — The probability of undersirable future consequences of actions (inaction) taken 
today. Thus, risk has a temporal element, and is a function of both probability and consequence. 

Program risk — The probability that the actual cost, time, or performance of a system will 
fail to match predictions. Also, the degree by which such predictions are missed and the associated 
consequences. 

Potential problem — An identified, but not yet occurring problem that if actualized, will 
impose unplanned resource demands, rescheduling, and/or degraded performance, quality, or safety 
margins. 

Risk area — A collection of related potential problems. Also, a common source of several 
potential problems. 

Potential problem analysis (risk identification) — Identification of risk areas and the 
sequence of interrelated potential problems that stem from them. Also can include identification of 
immediate cost, schedule, and performance impacts of potential problems, recognizing that potential 
problems may actualize at any one of several levels of severity and that there is a probability 
associated with each level. 

Risk assessment — Using the information from risk identification, and one or more quanti- 
tative techniques to synthesize the information, to create an overall assessment of program cost, 
schedule, or technical risk, and also an assessment of the risk contributed by each risk area. May 
include ranking risk areas by severity or timing in order to identify a course of action. 
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Risk management — Identifying alternatives, selecting an approach, and taking action in 
order to reduce risk to levels deemed acceptable by the organization. Action may be directed at 
risk reduction or in trading one type of risk for another. In some instances, work around plans and / 
or contingency budgets are defined as back-up solutions to the selected risk reduction approach. 

Probability — The relative frequency of an outcome of a repeatable, observable experiment. 
Also, a measure between 0 and 1 assigned to each outcome of an experiment based on its relative 
frequency. 

Subjective probability — A measure of the lack of information an organization or an 
individual has about the actual outcome of some future experiment. Essentially, it is a “degree of 
belief’ measure based on human experience and reasoning, as opposed to a “frequency of 
occurrence” measure. 

Probability encoding — A process whereby the lack of information of an expert is quan- 
tified as a subjective probability distribution on a state variable, developed under specific 
assumptions, in a scientifically correct way, with as much accuracy as is justifiable. Accuracy can 
be increased by spending more time per encoded variable, or by combining the opinions of several 
experts. 

Random variable — A math variable X mapping the outcomes of an experiment (perhaps 
non-numerical, such as coin flips resulting in HTH) onto real numbers, for example the number of 
heads in three consecutive coin tosses. 

Probability distribution — A mathematical formula that described how probability is 
assigned to the real numbers in the range of random variable. The distribution may be described by 
either a density function p(x) or a cumulative probability function F(x). F(x) simply cumulates the 
values of p(y) for all y < x. In the case of specific families of distributions (exponential, beta, 
normal, etc.), the distribution is completely specified once the values assigned to the parameters of 
the distribution are identified. 


III. ENCODING SUBJECTIVE PROBABILITY 


Most variables used as inputs to cost, schedule, and performance estimates range over a 
continuous scale (e.g., weight, time, voltage). Therefore, it is critical that the risk analyst have a 
process for converting each uncertainty estimate by an expert into an appropriate continuous prob- 
ability distribution. The key step in this “probability encoding” process is the conversion of several 
(typically, 3, 4, or 5) numbers elicited from the expert into the parameters of the specific distribu- 
tion being used. Of course, the choice by the analyst of distribution type (beta, normal, etc.) is a 
preliminary step to the elicitation of the estimates from the expert. 

The process of encoding expert option on uncertainty is a necessary part of any risk assess- 
ment. Without it, the only other source of probability information is historical data. Many NASA 
programs are to build one-of-a-kind systems, hence historical data is scarce and may not be directly 
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applicable. The risk analyst asks the technical experts with whom he interacts to draw analogies 
from their experience with similar programs. Thus, risk analysis is more closely related to estimat- 
ing by analogy than it is to estimating based on a historical data base or physical laws. 

The interview between risk analyst and technical expert should consist of the following five 

steps: 

1. Motivating 

2. Structuring 

3. Conditioning 

4. Encoding 

5. Verifying. 

The first three steps are referred to as pre-encoding, and are performed in order to make the 
encoding step as accurate and painless (for the expert) as possible. Pre-encoding can be very time 
consuming the first time that the analyst and expert sit down together. As the expert learns what is 
expected of him and how his estimtes will be used, pre-encoding time can be reduced by a factor 
of ten — say from 20 minutes to 2 minutes per variable being encoded. 

Motivating the expert includes discussing why point estimates of program variables need to 
be treated probabilistically, why subjective probability is a valid approach, and how his input will 
be used. Structuring is the analyst’s attempt to make the estimating job easier by using units of 
measure the expert is accustomed to, by using examples, and often by modeling the variable to be 
encoded. As an example of modeling, a cost variable may be estimated easier if it is broken into 
fixed and variable costs, or recurring and nonrecurring. A performance variable that is related via 
an equation or model to several technical variables may require the encoding of the technical 
uncertainty, then the use of Monte Carlo simulation (to be discussed later) of the model to obtain a 
probability distribution on performance. Conditioning of an expert eliminates bias, which is a con- 
scious or unconscious discrepancy between the subject’s responses and an accurate description of 
his underlying knowledge. The two most common causes of bias in subjective probability encoding 
are anchoring (lock in on most likely value and underestimate range of extremes) and unstated 
assumptions. Anchoring may be avoided by asking for extreme values first, and central values 
(mode or median) second. Other less common sources are availability (base estimates on most 
recent or most sensational memories) and coherence (base estimate on how many logical scenarios 
can be constructed. 

Accepted methods of probability encoding will be presented below. As mentioned earlier, 
the analyst typically selects the distribution type but may seek input from the expert in this selec- 
tion. The analyst then interviews the expert (questionnaires may also be used, if the expert and the 
analyst have an established relationship) and via use of the method converts expert opinion into 
parameters of the distribution. An important final step in the encoding process is verifying that the 
distribution actually matches what the expert believes. Three methods of verification are: (1) Show 
the expert a graph of the distribution, perhaps at a computer terminal. (2) Encode the cumulative 
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distribution function for the variable using a different method, specifically the bisection technique 
for distribution on a finite interval [a,b] and the fractiles technique for distribution on a half-line or 
the entire real line. Next, compare the cumulative distribution of the encoded variable with this 
newly generated cumulative curve for agreement and/or discrepancy. (3) Calculate several percentile 
points (say, 0.1, 0.3, 0.5, 0.7, 0.9) for the encoded distribution and present these to the expert to 
see if he agrees or disagrees with them. 


A. Encoding a Uniform Distribution 

The uniform distribution is a two-parameter distribution, the parameters being the left (a) 
and right (b) endpoint of the interval over which probability is uniformly distributed. A uniform 
distribution is appropriate for situations where requirements or perhaps physical constraints restrict 
the range of a variable to fall between two extremes with probability 1.0, but where the probability 
of any subinterval of length c (c < b-a) is equal to c/(b-a). The appropriate density function, given 
the expert has identified a and b, is: 


p(x) = l/(b-a), x in [a,b] , 

= 0, otherwise. 


B. Encoding a Triangular Distribution (Method 1) 

The triangular distribution has been a popular distribution among cost analysts because of its 
simple form. The three parameters needed to specify a triangular distribution are the endpoints (a 
and b) and the mode m, which corresponds to the peak of the triangle. Thus, by interviewing the 
expert and obtaining estimates for m, a, and b, the triangular distribution density function is: 


= 2(x-a) 

pw (b-a)(m-a) 


x in [a,m] 


2(b-x) 

(b-a)(b-m) 


x in [m,b] 


= 0 elsewhere . 


If it is desired to calculate the mean and variance of a triangular with parmeters a, b, and 
m, use: 


(ul - E(X) = (a -f m + b)/3 
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<t 2 = V(X) = [(b-a) 2 ) 4- (m-a) (m-b)]/18 . 


C. Encoding a Triangular Distribution (Method 2) 

Because engineers sometimes find it difficult to estimate the endpoints (a and b) of the 
triangular distribution, it may be better to ask for percentile points a' (a < a' < m) and b' (m < 
b' < b) and probabilities and p such that 


P(X>b') = s 

and 

P(x<a') - p . 

Note that s and p can be preselected by the analyst (e.g., s = p = 0.10) or selected 
arbitrarily by the expert being interviewed (p does not have to equal s). A major drawback of 
method 2 for triangular distribution encoding is that the following “accelerated Newton Raphson” 
recursive algorithm must be used to converge on the unknown values of a and b so that then the 
density function in method 2 is defined. Here h is an estimate of the height of the triangle. 

Step 1: Set h = (2/b'-a') as an initial estimate. 

Step 2: b — b' + (p/h) + (2b'p/h + p 2 /h 2 -2pm/h) 1/2 

Step 3: a — a' - (s/h) - (s 2 /h 2 - 2a's/h + 2sm/h) 1/2 

Step 4: f(h) = (V -a')h + (p + s — 2) + (2b'ph + p 2 -2pmh) 1/2 + (s 2 -2a'sh + 2smh) 1/2 

Step 5: f(h) - (b'-a') + (b'p-pm)/2b'ph + p 2 -2pmh) ,/2 + (sm-a's)/(s 2 -2a'sh-2smh) ,/2 

Step 6: f'(h) = —(b'p — pm)/(2b'ph + p 2 — 2pmh) 3/2 — (sm — a's)/s 2 — 2a'sh + 2smh) 3/2 

Step 7: h new = h-f(h)f(h)/([f(h)] 2 -f(h)f'(h)) 

Step 8: Set h = h new and return to step 2. 

D. Encoding a Beta Distribution 

The beta distribution defined on an interval [a,b] has been widely applied in program risk 
analysis. Although it is more technically complex than the triangular, it offers a wider variety of 
shapes (Fig. 2) and these shapes are controlled by two “shape parameters” y and r\. Thus, the beta 
is a four parameter distribution whose density function is fully known if a, b, y, and t] are known. 
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A four-part questionnaire is used to encode information about a beta distribution, following 
a technique developed by Stowell at Lockheed-Georgia Company. The expert is asked for best case 
(lowest, for weight, cost, or schedule) estimate “a,” a most likely estimate “m,” and a worst case 
(highest) estimate “b,” just as in triangular method 1 above. Also, he is asked to place a confi- 
dence level on the most likely ranging from “extremely confident” to “not confident,” as shown in 
Figure 3. Once these four values are known, a normalized modal value x is calculated according to 
x = (m — a)/(b-a). The normalized end-points are 0 and 1. 

Using this x and the confidence level in m, a “look-up” is performed using the matrix in 
Figure 4 to identify the appropriate shape parameters. Note that the shape parameters y and -q in 
this matrix range from 1 to 10. These limits are appropriate for program risk analysis because: (1) 
shape parameters greater than 1 assure distributions with exactly one mode; (2) if parameters 
greater than 10 are used, the distributions begin to have extremely peaked appearance and practical- 
ly zero probability assigned to 40 to 60 percent of the range [a,b], contrary to the intent of the 
expert. Note also that when y = r\, the distribution is symmetric and this occurs when m is 
selected to be situated roughly at the mid point of a and b. 


E. Encoding a Normal Distribution 

Although few engineering variables can actually range from — 00 to +o°, the normal dis- 
tribution is occasionally used in risk analysis. For example, many naturally occurring distributions 
are known to fit a normal distribution, such as the distribution of height and reach in humans, the 
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.... AVrfW AS P£G7“ 

FOR THE TECHNOLOGY MTtO WW6 . PLEASE ANSWER THE FOLLOWING 

QUESTIONS RELATIVE TO THE BENEFIT PARAMETER 

ASSUMING TECHNOLOGY AVAILABILITY IN THE YEAR 1115 

A. SMALLEST PERCENT IMPROVEMENT (OVER CONVENTIONAL) 

B. MOST LIKELY PERCENT IMPROVEMENT * (2 

C. LARGEST PERCENT IMPROVEMENT - !2 

0. CONFIDENCE IN MOST LIKELY BENEFIT (CIRCLE ONE) 


1L 


EXTREMELY 

CONFIDENT 


CONFIDENT 


SLIGHTLY 

CONFIDENT 



NOT 

CONFIDENT 


BETA DISTRIBUTION WITH 
SHAPE PARAMETERS 7-3 »7* 6 

1. 167 


11 13 18 

Figure 3. Conversion of questionnaire responses to beta distribution. 


DOMAIN OF 


SHAPE PARAMETER MATRIX 


NNORMALIZED 
\ MODAL 
RELATIVE \ VALUES 
PEAK \ 

HEIGHT \ 

(Confidence Ratingv 
for Modal Value) ^ 

0 

to 0.15 

0.15 
to 0.25 

0.25 
to 0.35 

0.35 
to 0.45 

0.45 
to 0.55 

0.55 
to 0.65 

0.65 
to 0.75 

0.75 
to 0.85 

0.85 
to 1.00 

HIGHEST 

y*2 

y= 3 

y=4 

y=6.5 

Y* 8 

y*8 

y=8 

y=8 

y=8 

(Extremely Confident 

0=8 

*7=8 

0=8 

0-8 

0=8 

0=6.5 

0=4 

0=3 

0=2 

HIGHER 

Y=\.5 

y=2.25 

y=3 

)*4.25 

y=6 

y=6 

y=6 

Y=6 

y=6 

(Very Confident) 

0=6 

0=6 

o=6 

0=6 

0=6 

0=4.25 

0=3 

0=2.25 

0=1.5 

HIGH 

K= 1.25 

y=l.75 

y=2.25 

y* 3 

y=4 

y=4 

y=4 

y=4 

y*4 

(Confident) 

' 7=4 

0=4 

0=4 

0*4 

0=4 

0=3 

0=2.25 

0=1.75 

0=1.25 

LOWER 

NA 

y= i.50 

y=i.75 

y*2.25 

y=3 

y=3 

y=3 

y=3 

NA 

(Slightly Confident) 

NA 

0 = 3 

0-3 

0=3 

0=3 

0=2.25 

0=1.75 

0=1.50 

NA 

LOWEST 

NA 

y=i.25 

y=i.5 

y=i.75 

y=2 

y= 2 

y=2 

y=2 

NA 

(Not Confident) 

NA 

1 1=2 

0=2 

»J*=2 

0=2 

0=1.75 

0=1.5 

0=1.25 

NA 












Figure 4. Table to identify appropriate beta shape parameters from 

four-part questionnaire. 




distribution of output of many manufacturing processes, etc. Also, the output of a Monte Carlo 
simulation is typically normal, regardless of the input distribution types and the complexity of the 
equation simulated. The output of a simulation may be an input to a risk analysis. 

The normal distribution is a two parameter distribution, where the parameters are the mean 
p and variance cr 2 . p and a 2 may be known from history, estimated from experimental sample 
data, or may be subjective inputs from one expert or perhaps an average of such inputs from 
several experts. In the case of subjective estimate of p and a 2 , it is important to remind the expert 
of the correspondences: 


p ± cr excludes 0.32 probability 
p ± 2a excludes 0.046 probability 
p ± 3a excludes 0.0027 probability 
p ± 4a excludes 0.00008 probability. 


F. Encoding a Weibull Distribution 

Occasionally an expert can specify a lower bound “a” on a variable, but feels that the upper 
end of the distribution should have a tail similar to the normal distribution. The appropriate dis- 
tribution to model this type of uncertainty is the Weibull. This distribution has been used exten- 
sively to model time-to-failure for electrical and mechanical components. The expert inputs a modal 
value “m” as before, but selects a high value “b” with a tail probability T equal to the probability 
that the actual value will exceed b (Fig. 5). Typically tail probabilities range from 0.05 to 0.20. 

The low estimate “a” is one parameter of the Weibull. The other two parameters are a scale 
parameter 8 > 0 and a shape parameter (3 > 0. When a — 0 and (3 — 1, the Weibull reduces to 
an exponential distribution with parameter 1/8. The density function is given by: 

p(x) = (3/8[(x — a)/8] p_l exp[ - (x - a)/8] p , x > a , 

— 0 , otherwise. 


The cumulative distribution function has the relatively simple form: 

F(x) = 1 - exp[ - (x - a)/8) p , x > a . 

A numerical algorithm is used to converge on the unknown parameters and : 
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• SPECIFY 


- LEFT ENDPOINT "A" 

- MOST LIKELY VALUE "M" 

- A RIGHT-HAND TAIL PROBABILITY "T" 

- A T-PERCENTILE POINT "B" 



• HANDBOOK CONTAINS A NUMERICAL METHOD TO CONVERGE 
TO THE SCALE AND SHAPE PARAMETERS OF DISTRIBUTION 

Figure 5. Encoding a Weibull distribution. 


Step 1: Compute constants 

K — — ALOG(T) 

Z = (b — a)/(m — a) 

Step 2: Set Ml = 1. 

M2 - 2. 

FM1 - -K. 

FM2 - 0.5*Z*Z — K. 

Step 3: If (ABS(M2-Ml).LE.EPSILON), (EPSILON - 1CT 5 ) 
Set (3 — M2 

and 1/8 = (1 — l/(3)*(m — a)**( — (3) 
else GO to 4. 


Step 4: Reset constants in Step 2 using: 

M3 = M2 — FM2*(M2 — M 1 )/(FM2 — FM 1 ) 
Ml - M2 
M2 = M3 
FM1 - FM2 

FM2 - (1 — I/M3)*Z**M3-K 
GO to 3. 
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IV. QUICK RISK ASSESSMENT METHODS 


When a risk analysis must be performed on short notice (within 1 to 2 days), extremely 
simple models must be used so that the analyst spends most of his limited time interacting with 
project personnel and collecting data. Dependencies between cost and schedule are time consuming 
to define and model. Therefore, the four methods presented in this section may be used to quantify 
risk (due to technical uncertainty) of meeting either a cost estimate, or a schedule milestone, but not 
both. Also, none of the methods presented here absolutely demand a computer for utilization. 
Calculations can be done by hand or hand-held calculator, which is an important feature if the 
analyst is away from his home office. A computer will make the calculations a bit faster, however 
the real benefit of a computer would be to use a spreadsheet to organize the data and perhaps plot 
simple graphs. 


A. Equi-risk Contour Method 

This method is oriented toward graphical depiction of risk and has been useful in capturing 
assessments of risk associated with several risk areas, provided by a team of technical experts. To 
illustrate the method, suppose the risk analyst has obtained from the project team a list of risk 
areas, and a description of one or more potential problems associated with each risk area. Each 
potential problem must be assessed for its risk, where risk is a function of both probability of 
problem occurrence and consequence of the problem given it has occurred. The consequence side 
of risk assessment has not been emphasized for several reasons: (1) often consequences are 
obvious; (2) consequences often can be dealt with by the PM internally; and (3) to include con- 
sequences in the risk assessment requires that the analyst encode the value system and risk attitudes 
of the PM. This enters into the specialty called “decision analysis,” which will not be discussed. 

In the equi-risk contour method, consequence may be measured in terms of “excess cost 
incurred” fixing a problem, or perhaps time delay or other operational penalty incurred to fix the 
problem. For example, suppose a problem has several levels of severity and could require any- 
where from $1000 to $1,000,000 to fix, if it occurs during some specific phase of the program. 
Further, suppose chances of this problem occurring range from 0 to 0.5, depending on which 
expert one interviews. The analyst should follow the following steps to quantify this risk: 

1 . Draw a risk assessment graph with consequence (cost) on one axis and probabilty of 
occurrence on the other as in Figure 6. 

2. Get the experts to agree on equi-risk contours and regions. The contours separate regions 
of different risk from each other. Typically three or four regions are specified: high risk, 
medium risk, low risk, and negligible risk. For example, if an expected loss of less than 
$100 is considered negligible risk by the team of experts (or by the project manager), 
then connect the points on the graph where (probability) X (consequence) — $ 1 00 to form 
the contour separating negligible from low risk, as in Figure 6. 
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COST TO 
CORRECT 
PROBLEM 
($ FY) 



Figure 6. Plotting expert assessments of risk on to a risk assessment graph. 


3. Ask the experts, either separately or through a group interaction technique, to rate each 
potential problem as to its probability and consequence. The results may then be plotted 
either as points or as risk regions. Once these data are displayed graphically (Fig. 6) it is 
easy to draw conclusions about expert agreement (disagreement), relative risk of one 
potential problem versus another, overall significance of one risk area relative to another, 
and so on. 


In summary, this technique is basic, easy-to-use, and focuses on information collection, 
group consensus, and graphical interpretation of risk. It is very well-suited to management briefings 
and should not be discarded simply because no high-level mathematics is involved. This technique 
is easily adapted to the situation at hand, be it program risk, manufacturing risk, mission opera- 
tions risk, etc. The analyst is able to adapt the definitions and scaling of probability and con- 
sequence to match the situation. The definitions should be agreed to in advance by the experts who 
will be providing the assessments. 
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B. U.S. Army Risk Factor Method (RFM) 

The method presented here is the simplest, but most subjective, of three methods developed 
in the mid-1970s by the U.S. Army and its contractors to perform program risk analysis. The other 
two methods, Probabilistic Event Analysis and the Probabilistic Network Model, will be discussed 
elsewhere. Each method permits a PM to develop a “Total Risk Assessing Cost Estimate 
(TRACE)” for use in RDT&E programming and budgeting. This concept emphasizes the allocation 
of funds to offset the effects of cost growth resulting from the occurrence of events that could not 
be programmed because of the lack of certainty that they would materialize. The concept 
recognizes that all funding demands arising during a development effort cannot be explicitly identi- 
fied in advance, however, it postulates that an aggregate of these demands can be predicted statis- 
tically and that this aggregate should be included in program and budget requests. 

The TRACE for a program is a specific point cost estimate, selected from within a range of 
potential costs for a material development program, defined to be the sum of two costs: 

1. The cost of specifically programmed activities. This cost corresponds to the RTD&E por- 
tion of the Baseline Cost Estimate (BCE). 

2. An additional amount sufficient to offset the adverse cost impact of additional, unplanned 
activities likely to require funding but not programmed/budgeted. 

The amount included in cost component 2 (called the TRACE Deferral) should not be 
interpreted as funding for all possible contingencies. Rather, it is based on PM judgment assisted 
by a formal, quantitative risk analysis to estimate (if possible) a probability distribution on the 
additional cost. The PM then applies his value system to decide if 50, 70, or 90 percent confi- 
dence of no cost overrun is acceptable risk. 

Potential problems included in the TRACE Deferral assessment include the following (a 
more extensive list of potential problems in RTD&E programs is included in Appendix 1): 

1 . Technical design changes to correct deficiencies or to accommodate nominal revisions in 
component performance. 

2. Rescheduling to work around technical problems and nominal budgetary limitations. 

3. Additional testing to verify design corrections. 

4. Additional hardware to support design modifications. 

5. Schedule slippages caused by late delivery of components or materials. 

6. Non-negligent human error. 

7. Program termination. 
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Note that requirements changes and effects of inflation are not included in the TRACE. The 
steps and an example of the Risk Factor Method (RFM) are now given. Both RFM and Proba- 
bilistic Event Analysis (PEA) are oriented toward the WBS elements of a program. RFM subjec- 
tively determines a risk factor for each WBS element where uncertainty exists and multiplies the 
associated cost by this factor to get an element of the cost risk. PEA goes into more detail on the 
potential problems, assessing what the problem will cost for an element as well as the probability 
of the problem impacting other elements, and the associated costs. 

The steps to employ in using the RFM are: 

1. Prepare the engineering cost estimate for each element as defined by the WBS. 

2. Compute a risk factor for each element in the WBS. This factor would include cost 
increases from two areas: 

(a) Cost increases which are due to the uncertainty in the individual element. 

(b) Cost increases which are due to external sources such as design changes resulting 
from problems with other elements. 

3. Multiply the risk factor from step 2 by the engineering cost estimate from step 1 for 
each element in the WBS. These new figures represent the revised estimate for each 
component. The sum of these revised estimates is the TRACE. 

4. Determine the TRACE for each year of the project. This is accomplished by multiplying 
the risk factor by each element’s portion of the BCE which is budgeted for each year of 
the program. 


RFM Example Problem 

Assume that a major hardware system is to undergo a modernization program. It is also 
assumed that this program will not involve any major advances in the state-of-the-art and little 
interaction is anticipated between the various program elements. Therefore, the RFM is applicable. 

1. The BCE is prepared for each element of the WBS as shown in column a, Table 1. 

2. Next, a risk factor is developed for each element of the WBS. The risk factor is a 
composite factor used to account for potential increases in element cost due to both 
internal and external effects caused by design changes as well as those costs of broader 
origin (e.g., modest work delays, delays in funding or obtaining parts) that can affect 
the cost of even the best managed programs. These factors could have been developed 
from historical data on similar systems or from subjective data. 
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TABLE 1 . TRACE TABULATION (Thousands of Dollars) 


b 

a Risk c — (a x b) 

WBS Element BCE Factor TRACE 


Component Development 

100 

1.10 

110 

Integration and Assembly 

20 

1.25 

25 

System Test and Evaluation 

25 

1.20 

30 

Total 

145 

N/A 

165 


3. The TRACE is now determined for the project by the multiplication of columns a and b. 
The total of column c is the total TRACE. 

4. The only remaining step is the allocation of $20,000 in TRACE deferral (i.e., the differ- 
ence between the TRACE and the BCE) to each year of the project. To accomplish this, 
the risk factor in column b of Table 1 is multiplied by the yearly expenditure of the 
BCE for each element within the WBS. The results are shown in Table 2. 

TABLE 2. YEARLY ALLOCATION OF RISK BY PROJECT YEAR 

(Thousands of Dollars) 


WBS Element 

Year 1 

Year 2 

Year 3 

Year 4 

BCE 

Trace 

BCE 

Trace 

BCE 

Trace 

BCE 

Trace 

Component Development 

50 

55 

30 

33 

20 

22 

100 

110 

Integration and Assembly 

0 

0 

8 

10 

12 

15 

20 

25 

System Test and Evaluation 

0 

0 

10 

12 

15 

18 

25 

30 

Total 

50 

55 

48 

55 

47 

55 

145 

165 


Advantage of RFM 

1. The analysis does not require a high analytical skill level. 

2. The analysis can be performed quickly and inexpensively in comparison to computer 
modeling. 

3. The analysis can be easily understood since complicated interrelations are avoided. 

4. The quality of the analysis can be easily evaluated by management due to the openness 
and simplicity of the analysis. 
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Disadvantages of RFM 

1. The most serious difficulty would appear to be the determination of the risk factors. 
Because of the apparent simplicity of the approach, there might be a tendency to use the 
risk factor as simply a “fudge factor.” To handle the risk factor in such a manner would 
not add to the credibility of the cost estimate. In addition, the risk factor is implicitly 
assumed to be constant for each element throughout the duration of the project. 

2. This type of analysis does not fully address the interrelationships between program 
elements; nor does the analysis address savings from the early completion of the program 
activities. These omissions adversely affect the quality of TRACE deferral determination 
and the allocation of the monies by fiscal year. This disadvantage varies with each 
project. It is more pronounced for the larger, more complicated systems. 

3. There is no indication of the total program risk. For example, in the networking 
approach, the TRACE is selected based upon a desired confidence level of successful 
completion. 

4. The TRACE computed by this method uses the BCE as a basis and is subject to the 
same bias as the BCE. 


C. U.S. Army Probabilistic Event Analysis (PEA) Method 

One of the difficulties in determining the risk factor in the Risk Factor Method is the 
interaction between program elements which can develop when one segment of a project gets into 
trouble. The TRACE PEA provides a method of accounting for overruns due to both the individual 
item and the external factors. Two type^ of problems are addressed for each element: a Type A 
problem which has a cost impact on a particular element alone and a Type B problem, which has 
cost schedule impact to the total project, excluding the Type A cost to the particular element. In 
describing this approach, reference is made to Figure 7. The following procedure is used: 

1 . The project should be broken down into WBS elements and critical milestones (column 

1 ). 

2. Estimate the point probability [P(A)j of a problem occurring in an element and causing 
an overrun in the element (column 2). 

3. Assess the cost impact of a Type A problem in the particular element (column 3). 

4. Given that a Type A problem occurs, estimate the probability [P(B/A)j of a cost impact 
on the remainder of the project (column 4). 

5. Find the unconditional probability of a Type B problem P(B) by multiplying column 2 
by column 4 (column 5). 

6. Assess the Type B cost and schedule impact upon the project (column 6). 
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Program 

Elements 

Prob. of 
Occurrence 

Cost Impact 
to Element/ 

Prob. of 

Coat /Schedule 

Date of Impact 

Expected 

Adjusted 

Occurrence 

Impact to Other 

Calendar 

FY 

Loss 

Expected 

P( A) 

Hi lea tone 

P(B/A) 

P(B) 

Program Elements 

Date 



Loss 

1 

2 

TYPE A 3 

4 

5 

TYPE B 6 

7 

e 

9 

10 

Vehicle 










1. Armor 

.20 

$3. 0M 


“ 

- 

4/78 

78 

$ .6M 

$ . 8H 

2. Suspension 

.60 

$ .AM 

1.0 

.60 

$I«, 27*1* 

6/78 

78 

$ . 84M 

$1. 12K 

3. Tracks 

.25 

S . 8H 

- 

- 

- 

6/78 

78 

S . 2M 

$ . 27M 

4. Power Train 

.60 

$I.5M 

.75 

.45 

$4H, 27*1* 

9/78 

78 

$2. 7M 

S3.6M 

5. Engine 

.20 

S . 5M 

1.0 

.20 

$ .5H, 1MN , 

Engine Qualifies- 

9/78 

78 

$ . 2M 

$ .2 7M 






tlon 





6. Integration 

.75 

$ . 2M 

1.0 

.75 

$2H, 2MI* 

11/78 

79 

S1.65M 

$2.2M 

Program Ml lea tones 










7, Initiate 

. 70 

0.0 

1.0 

. 70 

$6H, 3m* 

2/79 

79 

$4. 2M 

S5.6H 

Acceptance 

Testing 










8. OSD Review 

.60 

0.0 

1.0 

.60 

$2 . 5M, im* 

7/79 

79 

S1.5M 

$2M 

9. DSARC 

. 30 

0.0 

1.0 

.30 

$4M, 2m* 

10/79 

80 

$1.2M 

S1.6H 







Totals 

$13. 09M 

S17.46M 

♦Total Program Schedule Slips 







Figure 7. Probabilistic event analysis example. 


7. Predict the calendar time (column 7) and the fiscal year (FY) in which the problem 
would occur (column 8). 

8 . Calculate the expected loss for each element by adding the product of columns 2 and 3 
to the project of columns 5 and 6 (column 9). 

9. Calculate the adjusted expected loss (AEL) for each element. The AEL is a managerial 
adjustment factor to account for the fact that the PM will want to budget the full 
amount if it is felt that a problem and cost impact is virtually certain. For this example, 
the PM wants to fully fund for each element when the probability of cost impact is 
0.75 or higher. To obtain the AEL, divide the expected loss (column 9) by the AEL 
constant, 0.75 (column 10). 

10. Determine the total TRACE deferral by adding the AELs for all elements and mile- 
stones. Refer to Figure 8 to allocate the total TRACE deferral by FY. The objective is 
to program the TRACE deferral for the FY in which it will be needed. 

11. Determine the expected schedule slippage (column 3, Fig. 8) for each element by 
multiplying the probability of occurrence P(B) (column 5, Fig. 7) by the schedule 
impact (column 6, Fig. 7). 

12. Determine the expected schedule slippage prior to each nth element (column 4, Fig. 8) 
by totaling the expected slippage in the previous n-1 elements in column 3. 

13. Add column 4 to column 2 to find the probabilistic date of impact (column 5, Fig. 8). 
Fractional months are rounded down to minimize the impact of errors. 
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Program 

Date of 

Expected 

Expected Slippage 

Probabi listic 

FY 

AEL 



Elements 

Impact 

S lippage 

Prior to Given 

Date of Impact 




TRACE 


(Calendar 

(Months) 

Element (Months) 

(Calendar Date 




De ferral 

1 

Date) 2 

3 

4 

5 

6 

7 



Vehicle 









1. Armor 

4/78 

0.0 

0.0 

4/78 

78 

$0. 8M 

F 


2. Suspension 

6/78 

1.2 

0 . 0 

4/78 

78 

1. 12M 


> $2 . 19M 

3. Tracks 

6/78 

0.0 

1.2 

7/78 

78 

0. 27M 

JL 


4. Power Train 

9/78 

0.9 

1.2 

10/78 

79 

3.6M 



5. Engine 

9/78 

0.0 

2.1 

11/78 

79 

0.27M 

F 


6. Integration 

11/78 

1.5 

2.1 

1/79 

79 

2.2M 

7 

> S11.67M 

Program Milestones 
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7. Initiate 

2/79 

2. 1 

3.6 

5/79 

79 

5.6M 



Acceptance 

Testing 







~ 


8. OSD Review 

7/79 

0.6 

5.7 

12/79 

80 

2. 0M 

Y 

> S3.6M 

9. DSARC 

10/79 

0.6 

6. 3 

4/80 

80 

1.6M 

8 










LqJ 



Tot a 1 $17. 46 M 


Figure 8. TRACE deferral allocation. 

14. Determine the FY in which the probabilistic date of impact occurs (column 6, Fig. 8). 

15. Allocate the TRACE deferral to the appropriate FY. 

Advantages of PEA 

While this technique is more involved than the Risk Factor Method, it is still rather easy to 
use. The model addresses the interaction between elements and thus should give better results than 
the Risk Factor Method. 


Disadvantges of PEA 

1 . The technique is highly dependent upon the skill of the analyst to identify and account 
for various interdependencies and is sensitive to errors in subestimates. 

2. There is no indication of the program risk at the selected level of the TRACE. 

3. The TRACE computed uses the BCE as a basis and is subject to the same bias as the 
BCE. 
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D. Analytical Technique for Cost Risk Assessment 

An entirely different approach to cost risk assessment than the two above is presented in 
this section. Rather than working with the WBS, this method is based on the Cost Breakdown 
Structure (CBS) and assumes a collection of parametric estimating relationships is used to generate 
the RDT&E cost estimate. This technique is referred to as “analytic” because no Monte Carlo 
simulation is needed — the technique depends on statistical properties of cost estimating relation- 
ships (CERs) and sums of random variables (RVs). 

Assume that the total cost C upon which a probability distribution is sought may be written 
as a sum of element costs 


n 

C = 2 Yi 

i= 1 


Furthermore, assume each of these Y has a cost estimating relationship of the form 


Y - AX b 


where A and B were derived by linear regression using the equation 


InY = InA + B log X 


and, furthermore, a standard error (SE) value is available for the relationship. 

The basic precedure is presented then variations are discussed. The steps in the procedure 
■are as follows: 

Step 1 . Obtain an estimate of the mean jjl x and variance ct x for each independent variable X for 
which there exists uncertainty. The estimates of these parameters may be obtained direct- 
ly from historical data or an expert, or they may be obtained indirectly by first encoding 
the distribution on X and then using well-known formulas to calculate the mean and 
variance. 

Step 2. Estimate the mean variance of each cost element Y by using: 


M-v ~ A fx x B 
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<r y 2 = SE 2 + y + [(Ex + tr x ) B - ((Jix - <r x ) B ] 2 

Step 3. Assuming the cost elements Yi are stochastically independent, then calculate the mean 
cost jji c and the variance in cost <r c 2 using: 

n 

Pc pT) 

i=l 

n 

a c 2 = 2 <Ty. 2 

i=\ 1 


Step 4. By the Central Limit Theorem, the random variable C is approximately normal. There- 
fore, plot the cost distribution on C and/or calculate cost percentiles assuming C is 
normal with mean and variance as above. 

Variation 1 . Cost elements are not independent. The recommended approach is to recalcu- 
late the variance of C, but this time assume linear dependence (perfect correlation) among each 
pair of cost elements Yj and Yj. The appropriate formula to compute the variance in C under this 
(extreme) condition is simply: 

n 

o- c 2 = ( 2 (Jy) 2 
i = 1 


To see that this unusual-looking formula is valid, let Z = Y!+Y 2 . Then 


(J 2 2 - cry, 2 + ay, 2 + 2 [E(Y,Y 2 ) - E(Y,) E(Y 2 )] 


Recall that if the correlation between Y\ and Y 2 is 1, E(Y]Y 2 )-E(Yi)E(Y 2 ) = <x Y i a Y 2 . Thus 


cr z 2 = CTyi 2 + a Y 2 2 -T 2 cr Y i a Y2 = (<x Y i + a Y 2 ) 2 


If the risk analyst is uncertain about the nature of the correlation among the cost elements, 
calculating this second estimate of the variance of C gives him the opportunity to plot a second 
cost distribution, again having p c but with this larger variance than the independent case. It can be 


22 



stated to management that the cost risk falls between these two extremes. A more precise location 
of the cost risk curve involves a correlation matrix (see Variation 3 below). 

Variation 2. CERs contain a multiplicative complexity factor Q, which is itself a random 
variable and therefore a source of uncertainty in Yj. In this case, the only change in the four-step 
procedure involves Step 2, calculation of the mean and variance of Y. 

One has Yj = QAiX B i. The mean of Y, is easy because 


E(Yi) - Ci E(AjX Bi ) - CjAj E(Xi) Bi 


Assuming (for each i) that C and X are stochastically independent, 


Var(Y) - Var(C)E 2 (AX B ) + E 2 (C) Var(AX B ) + Var(C) Var(AX B ) 


(reverting to jjl and cr notation) = a c 2 Ap, x B 


+ (|jl c 2 + o- c 2 ) (SE 2 + A 2 /4 + [(p, x + a x ) B - (|x x - cr x ) B ] 2 ). 


Variation 3. Cost elements not independent but the correlation matrix is known. If the 
correlation matrix of the Yj variables is known, then the exact value of variance of C can be found 
using the following formula: 

n n-1 n 

Var(C) = 2 Var(Yi) + 2 2 2 Cov(Yi,Yj) , 

i = 1 i = 1 j = i + 1 


where 


Cov(Yj,Yj) — a Y iYj/°Y i0 Yj ■ 

This issue of where the correlation coefficients cry^ come from may be resolved in two 
ways. First, if the data base contains historical data on each Yj (actual) as it occurred on programs 
A, B, C, etc., then any statistical package will permit calculation of the matrix of correlations. The 
other alternative, used only if data is not available, is for the project personnel (not the analyst) to 
estimate the correlations among costs that can be expected for the program under analysis. 
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E. Analytical Technique for Schedule Risk Assessment 

The Program Evaluation and Review Technique (PERT) was developed by the Navy for the 
Polaris missile program in 1958 and was used in planning and evaluation of schedule uncertainty. 
Like the better known Critical Path Method (CPM), PERT utilizes a simple network model of a 
project to represent the precedence relationships among activities to complete the project. Network 
structures are clearly a better analytical framework for schedule analysis than the traditional bar 
(Gantt) chart, which most engineers use in their planning. The PERT/CPM network is simple in its 
node logic. Only one type of node is permitted in the body of the network: one that actuates only 
when all activities feeding in are completed, and once actuated all output activities start simultane- 
ously. Furthermore, only one start node and one terminal node are permitted. Also, no looping 
back to repeat an activity is permitted, in fact, no probabilistic branching may be modeled. 

Although PERT has many limitations (see Bibliography F), it is acceptable for a quick look 
at schedule risk on a development project — especially one where there is clearly one critical path 
and no “near critical” paths. Here is the PERT procedure: 

Step 1. Obtain three time estimates on each activity with uncertain duration: 

a — extremely optimistic time to complete 
m = most likely time to complete 
b = extremely pessimistic time to complete 

These time estimates assume resources are available to complete the activity, even if its 
duration extends beyond the planned time (m), but that no effort to accelerate the activity 
is being made. 

Step 2. Calculate a mean and variance in activity duration for each activity in the network using 
the traditional formulas: 


|x = 1/6 (a + b + 4m) 
a 2 - (b-a/6) 2 

These formulas assume activity time is beta distributed on [a,b], although they have been 
shown to be quite inaccurate as estimators of the true mean and variance. A better 
method is presented in Variation 1, below. 

Step 3. Using the mean completion time for each activity as input, apply the critical path algo- 
rithm (see any operations research text) to identify the project critical path. This path is 
the longest path from project start to finish. 
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Step 4. Calculate the mean project completion time by summing the mean times for activities 

along the critical path. Calculate the variance in project completion time by summing the 
variances of the activities along the critical path. This variance calculation makes the crit- 
ical assumptions that: (1) the activities on the critical path are independent; and (2) the 
variance along any near-critical path is less than the variance along the critical path. Both 
of these assumptions are typically doubtful for projects of the complexity as those 
performed by NASA. 

Step 5. According to the Central Limit Theorem, the project completion time will be approxi- 
mately normal and therefore a project time distribution may be plotted, and percentile 
points may be calculated using the standard procedures for normal distributions: 

Variance 7. In Step 2, rather than asking for the endpoints (a and b) of the hypothesized 
beta distribution on activity time, ask instead for the 0.10 and 0.90 percentile points. Then use the 
formulas due to Keefer (1981): 


p. 0.16m + 0.42[X(0. 1) + X(0.9)] 


a 


2 _ 


X(0.9)-X(0. 1) 2 
2.65 


These formulas have been proven to significantly outperform the classical PERT formulas. 


V. STANDARD RISK ASSESSMENT METHODS 


The methods presented in this section are referred to as “standard” because they are widely 
used in government and the aerospace industry. Two of the four methods presented have been in 
use in Program Planning at NASA-MSFC since 1978. The methods all utilize Monte Carlo simula- 
tion and therefore at least a microcomputer is needed, however they are not as sophisticated as the 
methods given in Section VI which require a specialized simulation package. Also, the methods are 
considered standard in that they require one to two weeks to apply (includes data collection, model 
runs, and interpretation of output) and this seems to be the amount of time typically allocated to 
conduct risk analysis for small programs such as aircraft mods, new shuttle payloads, etc. Of 
course large scale programs, such as a new fighter aircraft or the Space Station project, are of such 
complexity and dollar value that several months to a year should be devoted to each iteration of 
the risk analysis — and these may go on for several years. Finally, it is still true that the standard 
practice in industry and government is to separately perform cost risk assessment, schedule risk 
assessment, and technical risk assessment. The methods presented here are therefore not integrated 
methods, which are discussed in Section VI. 

The use of the Monte Carlo method (Fig. 9) to conduct a risk assessment is sometimes 
referred to as “uncertainty analysis.” As can be seen in the figure, a predictive equation or model 
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Figure 9. Monte Carlo uncertainty analysis. 

uses input variables that have been determined to be uncertain (random). A probability distribution 
on each of these random inputs is determined via methods as in Section III, or based on history. A 
random number generator is used to sample each of the distributions, and then an output of the 
model is calculated and this number is referred to as a “pass” through the simulation. A number 
(typically 100 to 300) of passes are made, thereby creating a sample on the output variables; the 
combined effect of the uncertainty in all the inputs is measured by the mean and variance of the 
output variable. Organizations typically have developed a general purpose Monte Carlo simulation 
program that accepts subjective probability distributions (often as they are encoded in interviews), 
other constants, and uses the model under study (e.g., a CER or entire cost model) as a sub- 
routine. For example, Lockheed-Georgia has a Monte Carlo Simulator called QUALM (Quantitative 
Uncertainty Analysis Model); NASA-MSFC has SAM (Stochastic Aggregation Model). Both of 
these models were developed in the 1980-81 timeframe. 

Before using Monte Carlo methods in conjunction with any model (e.g., a cost model, an 
aircraft performance model), it is critical that the analyst calibrate the model. This calibration 
process can be very time consuming and tedious, but it is necessary if the risk analysis outputs are 
to be valid. The calibration process consists of: 

1 . Remove any contingency figures from the baseline estimate being used by the project. 

2. Using the “most likely” values for each input variable, verify that the model produces 
the baseline estimate as output. If not, determine if a data entry error has been made. 

3. Using either a multiplicative or additive constant, calibrate the offending equations so 
that baseline inputs produce baseline outputs. 
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A. Simulation of Critical Path 

This method, used by PP02 at NASA-MSFC, is a modification to the PERT approach 
presented in Section IVE. Rather than analytically adding means and variances of activities along 
the critical path, this method convolutes the sequence of activity time distributions by means of 
Monte Carlo simulation. The critical path must be identified prior to the simulation, using either 
most-likely times on each activity or mean times on each activity as the input to the critical path 
algorithm. Two advantages of simulating the critical path are: 

1 . The encoding of time uncertainty on each activity is not restricted to the standard PERT 
3-point estimate. Any type of distribution may be used as long as: 

a. A method to calculate the parameters from the expert’s input exists. 

b. The simulation program to be used has a subroutine to generate random variates from 
distributions of the type selected. 

2. Simulation may pick up patterns of skewness in the input distribution and reflect these in 
the shape of the output histogram — the distribution on project completion time may be 
somewhat non-normal. 

The specific details of this method are: 

Step 1. Identify the project critical path. 

Step 2. For each activity on the critical path, obtain an estimate of the uncertainty in activity dur- 
ation using one of the methods of Section III. 

Step 3. Input each of these distributions to a Monte Carlo simulation program and simulate the 
equation Y - EXj, where X* is the random variable of time to complete activity i. 

Step 4. Analyze the output of the simulation (a sample on Y). 

Although this method saves time in that probability information is needed only for critical 
activities, it has several disadvantages and is therefore recommended only for panic situations. 

First, it assumes a single, known critical path that is determined based on input most likely times. 
In fact, most projects have multiple critical or near-critical paths. Furthermore, this method (like 
PERT) completely neglects the variances of near-critical paths, even though these may be larger 
than the variance along the critical path. Network simulation (presented next) is recommended to 
determine schedule uncertainty. 


B. Simulation of Project Network 

A network is the natural structure to graphically represent a project, where arcs represent 
activities (consume time, consume resources, generate performance), nodes represent junctures of 
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activities and starting points for other activities, and paths represent sequences of activities to get 
from one project milestone to another. The way in which the network’s arcs and nodes are con- 
nected represents project flow. PERT/CPM networks have a simple “node logic,” and this logic 
applies without exception to each node: all activities arriving at a node must complete before the 
node is activated (milestone achieved) and all activities leaving the node are simultaneously ini- 
tiated. More complex node logic is permitted the network-based simulation packages such as 
GERT, VERT, and RISNET. For instance, each of these permits probabilistic branching on the 
output side of a node, i.e., only one of several leaving arcs is activated and which one is acti- 
vated on a given pass through the project is controlled by a discrete probability distribution on the 
leaving arcs. Refer to the documentation for the specific package, or see the excellent 30-page dis- 
cussion by N. Whatley in the Cost Estimator’s Reference Manual. These packages are discussed in 
Chapter VI. 

Previous discussions of networks have assumed only time was modeled by the arc. 

Advanced network analysis also permits cost distributions to be specified for each activity, as well 
as cost dependencies with time. One of the network packages, VERT, has the capability to model 
performance generated by each activity. In this package, if a performance variable can be 
expressed as a mathematical function of other technical variables generated earlier in the project, 
then performance generated will be calculated as the simulation passes through that arc. Perform- 
ance generated may also be represented as a probabilistic branch, representing the various outcomes 
of some activity. These performance variables are also available as inputs to cost relationships later 
in the run. 

Integrated cost/schedule/performance risk analysis will be discussed in Section VI. This sec- 
tion refocusses on a network model of the project schedule. Various types of node logic may be 
embedded in the model, but a majority of the nodes are typically of the PERT/CPM type “AND- 
ALL.” Note that modeling all nodes in this way is a very poor choice for an R&D project. PERT/ 
CPM networks have no way to model the possibility that an activity will fail, that only one of two 
alternatives will be chosen, that as soon as success is achieved in one of several competing alterna- 
tives, the project will move on, etc. 

Interviewing technical experts and managers to define node logic is a time-consuming, but 
necessary part of network analysis. Establishing activity interrelationships and node logic has the 
beneficial effect of forcing: (1) better project definition; (2) recognition and acceptance of activity 
responsibility; (3) better overall planning and information for briefing higher authority. 

Whatever the node-logic, the process of network simulation (time uncertainty only) uses the 
following steps: 

Step 1 . Obtain assessments of time uncertainty on each activity in the network. Some organiza- 
tions stick to one distribution type (e.g., beta, triangular) while others permit any dis- 
tribution within the capabilities of the network simulation package being used. Also, if 
there are probabilistic branches in the network, each of these must have a discrete prob- 
ability distribution encoded, and input to the simulation. 

Step 2. Input distribution types and parameters. 
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Step 3. Run the simulation. Each pass through the network generates a point observation on 
project time. The collection of observed times constitutes a sample on project time. 

Step 4. Analyze the sample in Step 3 to determine characteristics of the project time distribution. 

Network simulation therefore differs significantly from critical path simulation in that the 
entire network is involved — all the project logic, and all the uncertainty information plays a role 
in generating the sample on project completion time. Much more confidence can be put in network 
simulation results than in either PERT or critical path simulation. 

Another significant feature of VERT and RISNET is that these packages automatically store 
and summarize information on “criticality.” Recall that in CPM/PERT there was one critical path. 

In network simulation, after each pass in the simulation, one can look back and identify which 
path was critical. By keeping track of these critical paths, both VERT and RISNET tell the analyst: 

1. How frequently did an activity appear on the critical path? For example, 0.50, 0.70, 

0.90, 0.99. Activities that appear with high frequency are the “most critical” and should 
be managed closely, and should receive first consideration for contingency funds to avoid 
delays in the real project. 

2. How frequently did a path serve as the critical path? 

NASA-MSFC has the ARTEMIS submodule “Probabilistic Analysis of Networks” (PAN) 
available on certain Hewlett-Packard computers. PAN assumes a PERT/CPM type network, and 
uncertainty distributions on activity time are simulted using Monte Carlo techniques. Time dis- 
tributions may take one of four forms: point, uniform, triangular, and normal. While PAN is 
schedule oriented, cost relationships to schedule duration may be modeled using the mathematical 
options available in the ARTEMIS system. PAN also produces an activity criticality index, as dis- 
cussed above. PAN has a relational data base that can be used to integrate input and output data 
into other ARTEMIS programs. However, use of ARTEMIS requires more computer sophistication 
than critical path simulation, or any of the techniques of Section IV. 


C. Simulation of Performance Estimating Model 

In the above, it was discussed how “performance generated” could be included in a network 
simulation. More often, estimates of performance uncertainty are generated completely independent 
of cost or schedule considerations. This performance risk analysis is more closely related to the 
sensitivity studies performed in conceptual and preliminary design than it is to project planning 
and control. Although the method for a single performance equation is illustrated below, it is easy 
to see that any predictive model for which there is uncertainty on inputs could be linked to a 
Monte Carlo simulator, and that probability information on the model outputs (estimates) could be 
generated. 

Consider the case of calculating the range of a strategic airlifter, flying at cruise altitude, 
with a given payload. The well-known Breguet range equation (Fig. 10) is used as the model to be 
simulated. The equation was calibrated by means of a multiplicative constant so that when nominal 
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1.0 


RANDOM VARIABLES: M. 



Figure 10. Uncertainty in aircraft range generated by Monte Carlo simulation. 


values for independent varibles are input, the nominal design range of the aircraft was output. A 
specific aircraft configuration was of course defined, and its range had been predicted by means of 
a sophisticated performance routine. This in no way precluded use of the simple range equation as 
a “sensitivity relationship” to be used for purposes of risk assessment. 

The four beta distributions shown in Figure 1 1 were elicited from respective experts serving 
on the design team. The weights engineer estimated weight empty (WE), the propulsion engineer 
estimated cruise specific fuel consumption (SFC), and so on. Note that no one individual had the 
knowledge to answer the question: “What’s the uncertainty in this aircraft’s range?” Inputting the 
parameters of these distributions and simulating the calculation of aircraft range led quickly to a 
histogram on range. These data were shown to be normally distributed, a sample mean and 
variance were calculated, and a smoothed distribution was fit as shown in Figure 10. Although the 
range scale has been removed, the requirement was set at 2400 NM and there was a 95 percent 
chance of meeting or exceeding this requirement. The mean range of this paper aircraft was 2550 
NM. Thus, uncertainty information on performance variables can be generated easily via Monte 
Carlo methods. 

With careful calibration of the model and valid input distributions, this approach provides 
useful information to the chief design engineer and PM on the range of possible performance 
outcomes due to technical uncertainty. Note that technical uncertainty is largely a function of tech- 
nology-level, and the engineers providing the subjective probability estimates on inputs were quite 
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CRUISE SPECIFIC FUEL CONSUMPTION 
(LB FUEL/HR/LB THRUST) 


Figure 1 1 . Beta distributions on the variables upon which range is dependent. 


aware of the technology levels on the specific configuration under study. Performance uncertainty 
curves such as in Figure 10 are extremely useful ways to compare competing design configurations, 
or the same configuration with different levels of technology. 


D. Simulation of Cost Estimating Model 

An analytical approach to cost risk assessment was presented in Section IVD. The simula- 
tion-based method presented here is used most often in cost risk assessment. Although some 
organizations simulate to assess strictly the impacts of uncertainty in inputs to the cost model, the 
approach that considers both input uncertainties and statistics-based uncertainties in the CERs 
themselves is presented. Cost models for system life-cycle-cost have been linked to Monte Carlo 
simulators. However, it is more common to find organizations (NASA-MSFC) that simulate to get 
at a probability distribution on RDT&E cost for a proposed project. The specific set of uncer- 
tainties that RDT&E cost risk assessment addresses are: 

• Design uncertainty (via CER input parameters). 

• Complexity factor uncertainty. 

• CER statistical uncertainties (confidence band in predicted cost expands as the input value 
deviates from the center of the data base). 

The steps in the process to build up a sample on a cost element Y ; are: 

Step 1. Obtain a probability distribution on the CER parameter X; and the complexity factor Q. 
Steps 2 through 5 apply to each simulation pass. 


31 



Step 2. Sample the distribution on X x to get a value. Calculate the corresponding point estimate 
of cost using AiXj B i. 

Step 3. Use the point estimate of cost (actu ally, this esti mates mean cost at X;), as a mean value, 
and a variance calculated by SE 2 VX^X'X^'Xi where X is a matrix made up of a 
column of Is, a column of historical X values. Sample this distribution under the 
assumption of normality and record the resulting cost. 

Step 4. Sample the complexity factor distribution and multiply the result with the cost in Step 3. 
This is recorded as the value of cost element Yj for the pass. 

Step 5. Record the cost elements Y x for the given pass. Assuming these elements are 
independent, sum them to get the estimate of total cost for the pass. 

These steps are illustrated in Figure 12. Dependence among cost elements can be built into 

the model as described in Section IVD. 


NASA-MSFC Cost Risk Model History 

Cost risk assessment has been performed at NASA-MSFC since 1978 using a series of 
computerized risk models: 

1978: NASA Headquarters (Econ, Inc.) Risk Model 

1980-81: MSFC Risk Model (Spartan, Inc.) 

Stochastic Aggregation Model (SAM) 

UNIVAC 1108, written in FORTRAN. 

1982: MSFC Risk Model (Revision B) 

Revised SAM 

Apple II Plus, written in BASIC. 

1984-: MSFC Risk Model (Revision C) 

Current: Some Sam Logic Retained, Triangular Method 2 to encode input probability, LOTUS 

1-2-3 based on IBM PC/AT. 


NASA-MSFC Cost Risk Application History 

Date Project 

5-79 Science and Applications Space Platform (SASP) 

5-79 Materials Equipment Carrier (MEC) 
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Figure 12. Monte Carlo cost risk assessment. 



11- 79 

12- 80 
5-81 
5-81 

5- 81 
7-81 
9-81 
1-82 
2-82 

12-82 

3- 82 

6- 83 

4- 85 


Atmospheric Cloud Physics Lab (ACPL) 
Centaur 

Solar Electric Propulsion Stage (SEPS) 
Tethered Satellite System (TSS) 

Gravity Probe-B (GP-B) 

Power System Platform (PSP) 

Acquisition Tracking and Pointing (ATP) 
Gravity Probe-B (GP-B) 

Gravity Probe-B (GP-B) 

Transfer Orbit Stage (TOS) 

Tethered Satellite System (TSS) 
Teleoperator Maneuvering System (TMS) 
Aeroassist Flight Experiment (AFE) 


E. Stochastic Aggregation Model (SAM) 

Although SAM is no longer available on NASA-MSFC mainframes, the model is presented 
here for three reasons: 

1 . It is a good example of how to build a comprehensive cost risk assessment simulation 
program. 

2. It is well documented (a Description and User’s Manual are available from Bill 
Ferguson) and it could easily be adapted to a microcomputer. 

3. Some of the SAM logic is employed in the current LOTUS-based procedure, for which 
limited documentation exists. 

SAM is a Monte Carlo simulation program designed to help the cost analyst quantify the 
uncertainty associated with a parametric cost estimate. The areas of cost risk which SAM will eval- 
uate are (1) cost estimating relationship (CER) independent variable uncertainty, (2) complexity fac- 
tor uncertainty, and (3) CER statistical uncertainty. 
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The input required by SAM is certain statistical data on the three areas of cost risk; output 
is a cost versus probability S-curve table which gives project cost as a function of the probability 
that the project can be completed for that cost or less. The output can be plotted as an S-curve 
with cost on the X-axis and probability or confidence on the Y-axis. 

SAM works by randomly sampling probability distributions provided by the user concerning 
the three areas of cost risk and using the resulting values to “cost the project out” several hundred 
times. By so doing SAM builds up the necessary statistical frequency data base to output the S- 
curve on what the expectations are regarding project cost. 

Here is what SAM requires for each cost line item on the SAM Input Form: 

Independent Variable XI (e.g., Weight, Power, 

SAM wants to know a ± 1 sigma (—70 percent) confidence range on what the value of the 
independent variable might be. The most likely value (ML) should generally be the baseline value 
used in the CER estimate (but see the additional discussion on this point below). The low (LO) 
and high (HI) values should represent the lowest reasonable value to expect and highest reasonable 
value to expect respectively, where. “reasonable” means “with 70 percent confidence.” SAM also 
wants to know a distribution type (T). For the independent variable, the distribution type can be 
any of the five distribution types now operational (Weibull, Triangular, Deterministic, Functional, 
or Thruput). 

Complexity Factor 

SAM wants to know the same kind of information about the complexity factor (i.e., a ± 1 
sigma range and distribution type). The most likely value should correspond to the baseline 
complexity factor used for this line item in the parametric estimate. The distribution type can be 
Weibull, Triangular, or Deterministic (but not Functional or Thruput). 

Other Multiplicative Factors 

This is an opportunity for the user to enter any other peculiar factors (such as an inflation 
factor) that were used in the parametric estimate for each cost line item. SAM simply takes these 
as given and multiplies all random samples of cost for the line item in question by the values 
given. This way, SAM is “Calibrated” to the parametric estimate (e.g., to the same year dollars). 

CER Coefficients 

Tell SAM the CER constant (A) and CER slope (Bl) from the Y - A(X1) B1 CER used for 
each cost line item. SAM will use this to estimate the most likely cost once SAM has randomly 
sampled the independent variable and complexity factor ranges. 

CER Statistics 

Here is where SAM accounts for the fact that CERs have statistical uncertainty associated 
with them. While SAM likes to know quite a bit about the CERs, it can get by on knowing just 
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the Standard Error (S.E.) and number of data points (N). However, SAM can do a better job of 
evaluating the statistical risk of a CER if the user can also provide the other three statistical facts 
called “optional” on the SAM Input Sheet. These are: 

- The mean of the independent variable data points (X). 

- The sum of the squared independent variable data points (XX 2 ). 

- The squared sum of the independent variable data points (EX) 2 . 

When the CER was originally derived (assuming a linear regression technique such as least squares 
was used), these three values were calculated. However, if they were not recorded, a companion 
computer program to SAM called “CER Statistics” can be used to generate these values (if the user 
is willing to input the indepedent variable data points into “CER Statistics”). See the attachment 
“CER Statistics Program.” 

Multivariable CER Input 

SAM will accommodate CERs up to four independent variables of the form: 


Y = A(X 1 ) B 1 (X2) B2 (X3) B3 (X4) B4 


If such CERs were used in the parametric estimate, this is where SAM wants to know the other 
independent variable values (X2 through X4) and the other slopes (B2 through B4). 

Distribution Types 

0 < T < 1 Weibull: 

The Weibull distribution has a high side tail extending beyond the given HI value which 
can be specified to hold some probability. Use the Weibull where there is some risk that the high 
value (HI) of the triangular distribution might be exceeded. Enter this high tail probability as a 
decimal fraction for T. Typically, Weibull tails are 5 to 20 percent probability (T = 0.05 to T = 
0.2). Anytime SAM encounters such a decimal entry for T it is assumed to indicate a Weibull dis- 
tribution and the value of T is used as the probability in the high tail. 

T = 1 Triangular: 

Use the Triangular distribution for most situations where the low, most likely, and high 
values of either the independent variable or complexity factor are known (70 percent confidence 
band). 
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T = 5 Deterministic: 


Enter T = 5 to signal SAM that a deterministic input is desired and enter the deterministic 
value desired as the Most Likely (ML) value. SAM will use the deterministic value as given and 
will not perform random sampling for this input. Use the deterministic for independent variables or 
complexity factors which have no uncertainty. 

T = 7 Functional: 

In most cost estimates there are “wraparound” cost line items which are functions of other 
cost line items (e.g., Project Management). These CERs use the sum of certain “above” line items 
as the independent variable. The T = 7 functional input can be used to make SAM sum the cost 
of certain above-line items (the cost as they have probabilistically been calculated) and use this as 
the independent variable for a wraparound CER. To signal a functional line item, enter T = 7 and 
specify a range of line items to be summed by entering the starting item number (M) in LO and 
the ending line item in HI. For example, if it is desired to sum the cost of line items 5 through 8 
and use the result as the input for a CER for line item 12, tell SAM this by entering for line item 
12 the following: LO — 5, HI — 8,T = 7. 

T = 8 “Thruput” Cost: 

Often it is desirable to “thruput” a cost number without exercising a CER and without the 
cost being affected by any random simulation process. Use T = 8 for any cost line item for which 
it is desired to thruput a cost and enter the thruput value as the independent variable ML. SAM 
will take the value entered in the ML position and use it as the final cost for this line item without 
any altering effects. The T = 8 Thruput input is only valid for the independent variable T input. A 
T — 8 Thruput for complexity factor is invalid. 


Summary of Valid Distribution Type (T) inputs 


Independent Variable 


Complexity Factor 


Weibull 

Triangular 

Deterministic 

Functional 

Thruput 


0 < T < 1 
T - 1 
T = 5 
T - 7 
T = 8 


Weibull 

Triangular 

Deterministic 


0 < T < 1 
T = 1 
T - 5 


Example Case (“Project X”) 

An example case for “Project X” has been made up to aid the user. The example has been 
designed to use all the features of SAM which are currently operational. 
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SAM Input Form for Project X: 

1 . Structures: Costed with a weight (lbs) versus cost CER. A probability distribution on 

weight was used for LO — 300 lb, ML — 500 lb, and HI — 900 lb, to be sampled as a 
Triangular distribution. A complexity factor Triangular probability distribution was also 
specified with LO = 0.9, ML = 1.0, and HI — 1.2. An inflation factor of 1.338 was 
specified to bring the CER costs from old year dollars to today’s dollars. No other 
calibration factor was required so Other = LO. The structure CER used was Y = 
0.58(500)°' 654 with an S.E. = 0.629, and N = 9. CER statistics were available for XL 
Since structure was costed with a single independent variable CER, the multivariable 
CER inputs are all zero. 

2. Attitude Control: Similar to above except a Weibull distribution was specified for the 

complexity factor with a 15 percent probability of exceeding the HI value of 1.4. 

3. Communications: Similar to above except that a bivariate CER was used [Y = 

0.01 1 (250) 1 269 (60)° °* 5 ] . 

4. Systems Test Hardware: The T = 8 signals a “Thruput” and SAM will use the 10 

entered as ML as thruput cost value (in this case, cost in millions) and will not perform 
any simulation around the $10M. Also T = 5 and ML — 1.0 for complexity factor 
signals a Deterministic complexity factor value of 1.0 is desired (again, without simula- 
tion). Since a thruput is being used SAM will ignore all CER entries (but zero’s are 
entered as a convenient way to cause the menu to progress during data file entry). 

5. Systems Test Operations: The T — 7, LO = 4, and HI = 4 signals SAM that this is 

a Functional cost line item (specifically that Systems Test Options is a function of line 
item 4— Systems Test Hardware). SAM will calculate the Systems Test Ops cost by using 
the simulated cost result of line item 4 as the independent variable input to the CER. 

Note that another feature of SAM is also illustrated here: the independent variable, 
instead of being input into a “regular” CER, is instead input into a CER of the form Y 

- 0.10(X1) 6 * * * 10 which is simply a way of modeling the case where Systems Test Opera- 
tions is calculated as 10 percent of Systems Test Hardware. 

6. Management: Again T — 7 signals a Functional line item. However, this time, since 

LO = 1 and HI = 5, SAM will sum the simulated costs of line item 1 through 5 and 

use the result as the independent variable input to the CER. 

SAM Output for Project X 

The output run for Project X consists of (1) listing of the input data, (2) the Project Cost 
Simulation results (a few hundred samples of total project cost), (3) a summary by cost line item 
of both (a) the average simulated cost and (b) the CER cost or “no risk” cost, and finally (4) a 
Confidence-Cost Table. 
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This last item, the Confidence-Cost Table, is the “bottom line” of the risk analysis. It gives 
the percent confidence associated with the given cost. For Project X, the CER estimate of $62M is 
only about 32 percent confident (68 percent chance that an overrun will occur). To get to a more 
reasonable level of confidence, say 70 percent, would require $132M (or about 113 percent contin- 
gency). This example is not typical of actual projects. Most of the “real” analyses performed by 
SAM will typically show that something like 20 to 40 percent contingency is usually sufficient to 
reach 60 to 80 percent confidence. 

SAM Technical Notes 

More on What the “ Most Likely” Value Means: 

For CER independent variable inputs, the correct value to input to SAM is the value used 
for the CER independent variable in the original baseline cost estimate less any contingency applied 
for risk. For example, if Electrical Power was estimated as a function of a watts requirement, say 
1150 Watts, and this 1150 Watts included 15 percent contingency in case the power requirements 
grew due to unforeseen circumstances, then the SAM Most Likely (ML) input should be 1000 
Watts. This is because the risk analysis itself is designed to evaluate the risk of a power growth 
and should be centered around a Most Likely that is risk free. However, care should be taken to 
try to distinguish between a contingency allowance for risk and a contingency allowance simply 
included to take care of miscellaneous small requirements. That is, if the extra 150 Watts were 
added to take care of requirements in the power budget that were not itemized but are known to be 
present, then the 15 percent is not a risk allowance and should be included in the Most Likely 
SAM input. 

How SAM Handles CER Risk: 

The risk of the CER is calculated by SAM as a confidence interval around the CER using 
the formula: 


Y ± (S . E . )(TDIST)(XTRAP) 


where Y = CER cost without risk, and the confidence limit is a ± band around the CER defined 
by the product of (1) the CER standard error (S.E.), (2) a Student’s Distribution factor (TDIST) 
for N-2 degrees of freedom and any, given percent confidence, and (3) a factor (XTRAP) which 
accounts for how far SAM is having to extrapolate beyond the mean of the data points of the 
CER. The value TDIST is generated internally by SAM as a function of N. XTRAP is generated 
internally as a function of the optional CER statistics on the SAM Input Form XI, XXI 2 and 
(XXI) 2 . 


XTRAP 



+ L + ( x p - x ) 2 

n (SX, 2 - (2X) 2 /n 
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If S.E. does not apply (e.g., if a CER was not used to estimate a cost line item or if the 
S.E. is unavailable) a zero entry may be made for S.E. In such a case, SAM automatically ignores 
all other CER statistics entries and will not sample a CER distribution. However, for data syn- 
chronization purposes, zero entries must be made for N, XI, XI 2 and (XI) 2 , X2, X3, X4, B2, B3, 
and B4. 

If S.E. is known, but the optional CER statistics are not available, enter the S.E., the 
number of data points N, and zero’s for the three optional CER statistics entries. 

If multivariable CERs were used, enter the independent variables (X2, X3, X4) and slopes 
(B2, 3, B4). For any unneeded entries (which will be all six values if a single independent variable 
CER was used), enter zeros. 

Interpretation of SAM Output: 

The cost risk data output by SAM should be interpreted as quantifying the cost uncertainty 

due to: 

• Uncertainty in the CER inputs (weight, power, etc.). 

• Uncertainty in the complexity factor 

• CER Statistical Uncertainty due to: 

— Scatter in the CER data points (S.E.). 

- CER sample size. The TDIST adjustment to S.E. allows for this risk by using a 

Student’s distribution for small N values. At high N values (>30 data points) the t dis- 
tribution approaches the normal — all of which SAM automatically accounts for when 
given a value for N from the SAM Input Form. 

— Degree of extrapolation beyond the mean of the CER data base (how well the “size” of 
the item being estimated matches the average of the CER data points). The XTRAP 
adjustment to the CER S.E. allows for this risk by causing the CER confidence in- 
tervals to spread as a function of the distance form the mean of the data. 

— Bias of Least Squares Log Transforms. Using the linear regression technique of least 
squares to derive a nonlinear y — ax b CER requires a log transform of the data. The 
curve fit is then made based on log values. When the values are converted back to 
antilog form a bias is introduced whereby the CER yields predicted costs that are less 
than 50 percent confident. The degree to which the CER is biased low is proportional 
to the S.E. but usually is on the order of 5 to 10 percent or so. SAM recognizes this 
bias by sampling the skewed log normal distribution for CER cost. Thus, the mean 
cost from the CER sampling will usually be higher than the CER estimate. 

There are obviously other risks which can impact cost other than those discussed above. 
Thus, SAM does not quantify all risks associated with a cost estimate. SAM does not account for 
such risks as: 
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- Design changes beyond those reflected by the ranges used for the CER independent vari- 
ables. 

- Additional requirements beyond those represented by the line items costed. 

- Risk associated with inappropriate design specifications, CERS, or complexity factors. 


VI. INTEGRATED PROGRAM RISK ANALYSIS 


The state-of-the-art in program risk assessment techniques is presented here. All techniques dis- 
cussed are network-based, and the activities in the network have both time and cost estimates. 
Furthermore, time and cost dependency is modeled and therefore the effects of these dependencies 
are included in each simulation pass through the network. The advantages of such integrated cost/ 
schedule simulation are: 

1. The cost uncertainty associated with the interaction of the activity schedules and their 
uncertainty is accounted for in the total cost. 

2. There is maximum flexibility in the selection of assumption and form of output in order 
to support the PM with “what if’ studies. Probabilistic information on costs, schedules, 
and project outcomes can be extremely useful for budgeting exercises. 

3. When maintained on a continuing basis, the project network is readily available for rapid 
response to questions. 

4. The network can be run with, then without, uncertainty information and fail-back alterna- 
tives in order to estimate the TRACE Deferral. 

5. The network can be used for tracking and control, too. 

Organizational factors that mitigate against the use of integrated cost/schedule risk analysis 
as depicted in Figure 13 are: 

1. Sophistication of techniques requires a dedicated specialist — M.S. -level with good 
computer, systems analysis, and people skills. 

2. Matching WBS to scheduled activities is challenging. Network logic is more complex, 
and the output of simulation can be very sensitive to the logic employed. Especially 
important are branching probabilities. 

3. Planning, conducting, and validating the data gathering are time consuming. Program 
management must be prepared to wait, perhaps several months, before the first useful 
output appears. 
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• CAN BE IMPLEMENTED USING GERT, VERT, RISNET, SLAM 

• BUILDING NETWORK, COLLECTING DATA IS TIME-CONSUMING 

• ONCE DEVELOPED, IS VERSATILE TOOL 







$ 


Figure 13. Integrated cost/schedule risk assessment. 






4. Cost and schedule analysis/control are often handled by separate organizations. The risk 
analyst must cross these boundaries. Also, the risk analyst is sometimes regarded as 
attempting to usurp the roles of schedule and cost analysts. 

5. Special software will have to be obtained, and analysts trained to use it. Such software 
packages exist; the three best known (GERT, VERT, RISNET) will be discussed only 
briefly below because ample documentation (textbooks, user’s guides) exists. 


A. GERT (Graphical Evaluation and Review Technique) 

GERT is a network simulation package (FORTRAN IV) available from Pritsker and Asso- 
ciates that includes features such as probabilistic branching, network looping back to previously 
activated nodes, multiple sink nodes, and multiple node realizations (repeat events), all of which 
are not available in PERT/CPM network models. These GERT features provide the user with the 
capability to model and analyze projects and systems, such as queues in manufacturing processes, 
of a very general form. Since many real-world projects and systems involve probabilistic 
occurrences, talse-starts, activity repetition, and multiple outcomes, GERT and its advanced version 
Q-GERT is an ideal tool for modeling and analysis. GERT is oriented toward supporting 
management in preliminary planning, and special studies, rather than toward project control. The 
capabilities of GERT are a subset of the capabilities of the general-purpose simulation language 
SLAM, also from Pritsker. SLAM is being used at NASA-MSFC by J. Steincamp and D. Lanier 
in PD34. 

In GERT, an activity has a probability associated with whether it is included in the project 
on a given simulation pass. This probability is an input to the simulation and is attached to one of 
the branches of the node of origin of the activity. For example, activity X has a 0.7 probability of 
inclusion shown in Figure 14. Also there are ten options for the time distribution to be encoded for 
an activity. Finally, each activity has a cost function associated with it made up of a fixed cost 
(incurred each time the activity is conducted) and a variable cost expressed as dollars per unit time. 

GERT nodes have both an input and an output logic. For inputs, the number of completed 
activities feeding a node required before the node activates can be specified. Furthermore, as 
shown in Figure 14 on node 16, a different number of completions may be specified for the first 
pass (3) than subsequent passes (2). Multiple completions fo the same activity count. Node 18 in 
Figure 14 illustrates another type of input logic — here the specified number of different activities 
must complete before the node activates. Options also exist that halt all other on-going activities 
once the input criteria are satisfied. Using these node options, complex substructures as shown in 
Figure 15 are easily prepared. 

One of the unique features of GERT is that it permits collection of statistics on how fre- 
quently each of several terminal nodes is realized. For example, one node in an R&D project 
might be “project wash-out.” A simulation might discover that there is a 0.25 chance of project 
wash-out, and then management could compare the cost and time distributions on that node to 
decide if the risk of failure was too great to continue the proposed approach. 
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Statistics can be collected for any designated nodes (project milestones) and the sink nodes 
on cost, schedule, and activity counts. For each of these nodes, information is provided regarding 
the mean, standard deviation, maximum, and minimum for both time and cost. These output data 
will be illustrated in the example below. 

The example R&D process shown in Figure 16 represents four R&D projects of identical 
structures, with a complex interlocking structure. Furthermore, two teams are involved. Research 
Team 1 starts on Project 1 (node 10) at the same time that Team 2 starts on Project 4 (node 40). 
When Team 1 terminates Project 1, either by washout at node 17 or successful completion at node 
17, it proceeds to Project 2 (node 20). Team 2 follows a similar process as it works Project 4, 
then Project 3. Also if either team completes its second project before the source node of the 
opposing team’s second project has activated, that team will begin work on a third project. 

The GERT inputs for this model are shown in Figure 17. Figure 18 shows the relative 
frequencies of success/failure for each project. Summary tables on cost and time for each project 
are available in the 1977 paper by Moore and Taylor. The statistics for the overall R&D activity 
are shown in graphical form at the bottom of Figure 18. Such data could obviously be used in con- 
tract negotiations and budget/manpower planning. 


B. RISNET (Risk Information System and Network Evaluation Technique) 

RISNET is a network simulation package that has been employed extensively on U.S. Army 
programs during the past 10 years. It has been developed and continually improved by John M. 
Cockerham and Associates, Inc. RISNET can run in either deterministic or probabilistic modes. 

Like GERT, RISNET represents cost as a function of activity time using a fixed cost plus a vari- 
able cost. However, RISNET permits these costs to have probability distributions (normal, uniform, 
or triangular) on them, or to be represented as a constant. Each activity can have up to 20 different 
funding categories (subcosts) associated with it, each with its own linear cost relationship. 

RISNET node logic is not as complex as that of GERT, described above. The only advance 
of RISNET node logic over PERT/CPM is that it permits probabilistic branching on the output 
side. RISNET has an interactive, menu-driven input module that makes data entry easy. Most 
analysts prefer the computer system’s data editor over the RISNET data editor. 

A real strength of RISNET is that it was developed specifically for application in the U.S. 
Army R&D area, so it is oriented toward fiscal years and the TRACE concept (discussed in Chap- 
ter IV). The RISNET output module produces graphics terminal output, printer output, and plotter 
output for deterministic and probabilistic simulations. Output is available by Fiscal year or pre- 
selected time interval (e.g., a quarter). Outputs similar to GERT include cumulative distributions 
and frequency distribution, for both time and cost by total program or preselected subnetworks. 
TRACE output is by Fiscal year, and includes the BCE, the TRACE Deferral, and the total 
TRACE, with probability distributions so that the PM can select his level of acceptable risk or tell 
others the risk implications of budget cuts. 
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Time Cost 


Activity 

(Nodes) 

Activity 

Description (Project) 

Probability 
of Occurrence 

Estimates (Days) 
Min Mode Max 

Distribution 

Estimates (S) 
Set-Up Variable 

10-11 

Project Start (1) 

1.00 


0 


Constant 

10,000 

0 

11-12 

Problem Definition (1) 

1.00 

20 

30 

50 

Beta 

0 

450 

12-13 

Research Activity (1) 

0.80 

60 

100 

120 

Beta 

2,000 

300 

12-11 

Redefine Problem (1) 

0.20 


0 


Constant 

0 

0 

13-14 

Solution Proposal (1) 

1.00 

7 

15 

20 

Beta 

0 

500 

14-15 

Develop Prototype (1) 

0.60 

60 

80 

100 

Beta 

7,000 

600 

14-12 

More Research (1) 

0.10 


0 


Constant 

0 

0 

14-11 

Redefine Problem (1) 

0.10 


3 


Constant 

0 

0 

14-17 

Project Washout (1) 

0.20 


0 


Constant 

1,000 

0 

15-16 

Implementation (1) 

0.60 

75 

90 

130 

Beta 

3,000 

500 

15-15 

Redevelop Prototype (1) 

0.40 

60 

80 

100 

Beta 

1,000 

600 

16-20 

Completion (l)/Project (2) 

1.00 


10 


Constant 

2,000 

0 

17-20 

Washout (1)/ Project (2) 

1.00 


10 


Constant 

2,000 

0 

20-21 

Project Start (2) 

1.00 


0 


Constant 

8,000 

0 

21-22 

Problem Definition (2) 

1.00 

7 

21 

28 

Beta 

0 

525 

22-23 

Research Activity (2) 

0.60 

25 

40 

90 

Beta 

2,500 

425 

22-21 

Redefine Problem (2) 

0.40 


0 


Constant 

0 

0 

23-24 

Solution Proposal (1) 

1.00 

15 

25 

40 

Beta 

0 

450 

24-25 

Develop Prototype (2) 

0.60 

20 

60 

90 

Beta 

5,000 

525 

24-22 

More Research (2) 

021 


0 


Constant 

0 

0 

24-21 

Redefine Problem (2) 

0.10 


3 


Constant 

0 

0 

24-27 

Project Washout (2) 

0.03 


10 


Constant 

1,000 

0 

25-26 

Implementation (2) 

0.80 

15 

40 

60 

Beta 

4,500 

375 

25-25 

Redevelop Prototype (2) 

020 

20 

60 

90 

Beta 

1,000 

525 

26-30 

Completion (2)/Project (3) 

1.00 


10 


Constant 

2,000 

0 

27-30 

Washout (2)/ Project (3) 

1.00 


10 


Constant 

2,000 

0 

30-31 

Project Start (3) 

1.00 


0 


Constant 

18,000 

0 

31-32 

Problem Definition (3) 

1.00 

35 

70 

120 

Beta 

0 

525 

32-33 

Research Activity (3) 

0.50 

80 

150 

200 

Beta 

4,500 

425 

32-31 

Redefine Problem (3) 

0.50 


0 


Constant 

0 

0 

33-34 

Solution Proposal (3) 

1. 00 

21 

40 

50 

Beta 

0 

625 

34-35 

Develop Prototype (3) 

0.55 

105 

170 

210 

Beta 

6,500 

600 

34-32 

More Research (3) 

0.10 


0 


Constant 

0 

0 

34-31 

Redefine Problem (3) 

0.05 


3 


Constant 

0 

0 

34-37 

Project Washout (3) 

0.30 


0 


Constant 

2,000 

0 

35-36 

Implementation (3) 

0.75 

60 

120 

200 

Beta 

5,500 

500 

35-35 

Redevelop Prototype (3) 

0.25 

105 

170 

210 

Beta 

1,000 

600 

36-20 

Completion (3)/Project (2) 

1.00 


10 


Constant 

2,000 

0 

37-20 

Washout (3)/Project (2) 

1.00 


10 


Constant 

2,000 

0 

40-41 

Project Start (4) 

1.00 


0 


Constant 

13,500 

0 

41-42 

Problem Definition (4) 

1.00 

20 

45 

60 

Beta 

0 

475 

42-43 

Research Activity (4) 

0.90 

21 

30 

55 

Beta 

3,500 

500 

42-41 

Redefine Problem (4) 

0.10 


0 


Constant 

0 

0 

43-44 

Solution Proposals (4) 

1. 00 

14 

30 

45 

Beta 

0 

625 

44-45 

Develop Prototype (4) 

0.70 

50 

90 

150 

Beta 

4,000 

500 

44-42 

More Research (4) 

0.18 


0 


Constant 

0 

0 

44-41 

Redefine Problem (4) 

0.10 


3 


Constant 

0 

0 

44-47 

Project Washout (4) 

0.02 


0 


Constant 

1,000 

0 

45-46 

Implementation (4) 

0.60 

35 

75 

100 

Beta 

2,700 

550 

45-45 

Redevelop Prototype (4) 

0.40 

50 

90 

150 

Beta 

1,000 

500 

46-30 

Completion (4)/ Project (3) 

1.00 


10 


Constant 

2,000 

0 

47-30 

Washout (4)/Project (3) 

1.00 


10 


Constant 

2,000 

0 


Figure 17. Activity descriptions with probability, time, and cost estimates. 
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R&D activity simulation. 





The RISNET computer model hardware requirements are the following: 

1. Hewlett-Packard 1000/A900 computer 

2. 3.84 megabyte (mb) memory 

3. 400 lines per minute printer (model 2608S) 

4. 1600 bpi magnetic tape drive (model 7970E) 

5. 65 megabyte (mb) fixed disk drive (model 7912) 

6. 404 megabyte (mb) removable disk (model 7935) 

7. Color graphics terminal (model 2627 A) 

8. Plotter (model 9782T). 

Software requirements are the following: 

1 . RTE-A with VC and REV 2326 

2. Graphics/ 1 000- 1 1 REV 2326 with both device independent graphics and advanced 
graphics. 


C. VERT (Venture Evaluation and Review Technique) 

The VERT network analysis model was developed by Moeller while employed by the U.S. 
Army. The model is public domain, written in FORTRAN for an IBM machine. An enhanced ver- 
sion of VERT is available from TRIAD Microsystems. Like the RISNET models, VERT is a 
stochastic simulation network model with probabilistic node logic. The VERT model is more math- 
ematically oriented than the RISNET models. VERT has 37 mathematical transformations which 
can be used to express the relationship between key variables, whereas the RISNET models have 
one linear relationship between time and cost. 

VERT also has the added advantage of incorporating performance values as a risk factor in 
network evaluation. With VERT an analyst can model time, cost and performance values, and 
uncertainties for each activity independently of each other using one of the following 1 6 different 
statistical distributions available in VERT: 

Beta 

Binomial 

Chi Square 

Constant 
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Erlang 

Exponential 

Gamma 

Geometric 

Hypergeometric 

Lognormal 

Normal 

Passcal 

Poisson 

Triangular 

Uniform 

Weibull 

Activity time, cost, or performance values can be modeled as a function of any other arc’s 
or node’s time, cost, or performance parameter in the network using any of the following 37 math- 
ematical transformations or a combination of these transformations: 


Code Number 

Transformation 

Restrictions 

Notes 

1 or 51 

X*Y*Z 


R 

(*Means multiply) 

2 or 52 

(X*Y)/Z 

— 

R Z NE 0.0 

(NE means not equal) 

3 or 53 

X/(Y*Z) 

= 

R Y*Z NE 0.0 


4 or 54 

1/(X*Y*Z) 

— 

R X*Y*Z NE 0.0 


5 or 55 

X + Y + Z 

=: 

R 


6 or 56 

X + Y + Z 

= 

R 


7 or 57 

X-Y-Z 

— 

R 


8 or 58 

-X-Y-Z 

— 

R 


9 or 59 

X*(Y + Z) 

— 

R 


10 or 60 

X*(Y-Z) 

= 

R 


1 1 or 61 

X/( Y + Z) 

= 

R Y + Z NE 0.0 


12 or 62 

X/(Y-Z) 

= 

R Y-Z NE 0.0 


13 or 63 

X*(Y) Z 

— 

R Y GT 0.0 

(GT means greater than) 

14 or 64 

X*(LOG e (Y*Z) 

= 

R Y*Z GT 0.0 


15 or 65 

X*(LOG 10 (Y*Z) 

= 

R Y*Z GT 0.0 


16 or 66 

X*(SIN(Y*Z) 

— 

R 


17 or 67 

X*(COS(Y*Z) 


R 


18 or 68 

X*(ARCTAN(Y*Z) 

— 

R 


19 or 69 

X GE Y- — Z 

= 

R 

(GE means greater than or 


X LT Y-----Y 

= 

R 

(LT means less than) 

20 or 70 

X GE Y Y 

- 

R 



X LT Y Z 

= 

R 
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21 or 71 

X GE Y Z 

= 

R 


X LT Y— -X 

= 

R 

22 or 72 

X GEY— -X 

— 

R 


X LTY Z 

— 

R 

23 or 73 

(X* Y) + Z 

— 

R 

24 or 74 

(X*Y)-Z 

= 

R 

25 or 75 

(X/Y) + Z 

= 

R Y NE 0.0 

26 or 76 

(X/Y)-Z 

= 

R Y NE 0.0 

27 or 77 

(X + Y)*Z 

— 

R 

28 or 78 

(X + Y)/Z 

= 

R Z NE 0.0 

29 or 79 

(X-Y)*Z 

= 

R 

30 or 80 

(X-Y)/Z 

= 

R Z NE 0.0 

31 or 81 

X + (Y*Z) 

— 

R 

32 or 82 

X-(Y*Z) 


R 

33 or 83 

X + (Y/Z) 

= 

R Z NE 0.0 

34 or 84 

X— (Y/Z) 

= 

R Z NE 0.0 

35 or 85 

-X-Y + Z 

= 

R 

36 or 86 

-X + Y + Z 

— 

R 

37 or 87 

X/Y/Z 


R Y NE 0.0 


Z NE 0.0 

Transformation numbes 1-37 and 51-87 use floating point computations initially to derive a 
value for R. However, transformations 51-87 truncate R to an integer value whereas transformations 
1-37 retain R in its floating point form. 

Structuring a mathematical relationship within a VERT network consists of essentially the 
following three phases: 

1 . Long or complicated mathematical relationships need to be broken down into a series of 
three- variable unit transformations shown previously. 

2. Values for each of the three variables X, Y, and Z in each single unit transformation 
must be defined. These values can be retrieved from (1) previously processed arcs or 
nodes, (2) constants entered on these satellite arc records, or (3) one of the previously 
processed transformations computed in the current series of transformations used to 
generate a time value for the current arc under consideration. Values calculated for each 
time transformation are consecutively, temporarily stored in a one-dimensional array. 

This enables retrieving the value calculated for a prior transformation for use in the 
current unit transformation. On completion of all the unit time transformations for a 
given arc, this temporary storage array is cleared. Thus, only the values calculated for 
previously derived unit time transformations developed for the current arc under consider- 
ation can be referenced. When retrieving numerical values from a previously processed 
arc or node, the time, cost, or performance value calculated for the referenced node or 
the primary (not cumulative) time, cost or performance value generated for the refer- 
enced arc is retrieved. 
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3. Results of each of the unit transformations needed to develop a value for an arc’s time 
can be either summed into the overall time value generated for the arc under considera- 
tion or it can be omitted. When the resulting value of a unit transformation is omitted, 
this transformation is generally an intermediate step for calculating the value of a long or 
complicated mathematical relationship. 

Transformation Example. Suppose the value for the performance of a given arc is related 
to the time, cost, and performance values generated on this arc and other previously processed arcs 
and nodes as follows: 


PA10 


(PA1 + PA2 + PA3) 
(PA4 * PA5 * PA6) 


(TA1) * 


(LOG e (CAl 


* CA2)) 


+ (188.6) 


*(TA10) 

(CA10) 


+ (15.8) * (TNI) 


where 


TNI — the time value for the node N1 
TA1 — the time value for the arc A1 
TA10 = the time value for the arc A 10 
CA1 — the cost value for the arc A1 
CA2 = the cost value for the arc A2 
CA10 = the cost value for the arc A 10 
PA1 = the performance value for the arc A1 

PA2 — the performance value for the arc A2 

PA3 = the performance value for the arc A3 

PA4 = the performance value for the arc A4 

PAS = the performance value for the arc A5 

PA6 = the performance value for the arc A6 

PA 10 = the performance value for the arc A10 
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The following dimensioned record layout illustrates how the preceding equation is put into 
record form. 


A10 

RPERF 1 50 PA1 

PA2 

PA3 

Trans. No. 1 

A10 

RPERF 2 40 PA4 

PA5 

PA6 

Trans. No. 2 

A10 

RPERF 3140TA1 

CA1 

CA2 

Trans. No. 3 

A10 

RPERF 4 IS 1.0 

2.0 

3.0 

Trans. No. 4 

A10 

RPERF 5 2SK188.6 

TA10 

CA10 

Trans. No. 5 

A10 

RPERF 6 1SK15.8 

TNI 

K1.0 

Trans. No. 6 


VERT allows the user to tie together mathematically any two points within the network, as 
well as to establish a variety of mathematical relationships between time, cost, and performance 
values for any given activity. These features are not found in any other network simulation tool. 
Thus, the VERT model is the appropriate tool for the analyst who is serious about performing 
integrated cost/schedule/performance risk assessments of R&D programs. TRIAD VERT has the 
additional capability of presenting the probabilistic output of VERT by fiscal year and by month. 

Neither VERT nor TRIAD VERT has the color graphics output capability of RISNET; both 
VERTs are printer-oriented. Although VERT does not require a high degree of computer sophis- 
tication to operate, its input process is a batch-oriented, 80-column record format, so it is more 
time consuming and tedious to use than RISNET. VERT is written in FORTRAN IV and has been 
adapted to IBM, Univac, DEC VAX 11/780, CDC, and PRIME computers. For a case study 
application, see the paper “Operational Planning with VERT” by Moeller and Digman. 


D. A Conceptual Model for Integrated Technical/Cost/Schedule Risk Assessment 

Very little has been said in this handbook about the risk identification phase of risk analysis 
(Fig. 1). Risk identification is a period of intense interaction between risk analyst and project 
personnel to define the nature of the risk and how/when it might come to bear. The best dis- 
cussions in the literature on this topic are the 1982 paper by Kramer (describing his work at 
Boeing-Vertol) and the 1982 paper by Batson, “Impact Diagrams: A Graphical Procedure for 
Potential Problem Analysis,” presented at the San Diego ORSA/TIMS meeting. 

Following the work of Kramer, a concept for linking risk identification data gathering to 
cost/schedule network simulation was proposed by Batson at Lockheed-Georgia in 1983. Kelly 
developed the concept into a well-defined process (called CPM/RISK) which is outlined here. The 
essence of the process is that information on risk areas and their time/cost impacts are collected 
from the design and technology engineers, using a form as shown in Figure 19. Once the impact 
points of all risk areas on the network are known, it is then necessary to reorganize potential 
problem information by grouping according to activity (Fig. 20). This summary sheet would then 
be shown to the person who would manage that activity on the project, and he could alter it to 
match his judgment about probabilities, penalties incurred, etc. Note also that he is permitted to 
specify a probability and cost for an “unknown” problem, i.e., a problem not listed. Assuming this 
individual has managed a similar activity before, he ought to be able to estimate the extent of 
unknowns that lurk in the future. 
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On 

Activity: _ 

Problem Number i . ... .. 
Probability of Occurrence: 


PROBLEM 

DESCRIPTION 

OPTIMISTIC 

MOST LIKELY 

PESSIMISTIC j 

Probability of 
Occurrence 

Weeks To 
Resolve 

Manhours 
To Resolve 

Probability of 
Occurrence 

Weeks To 
Resolve 

Mon hours 
To Resolve 

Probability of 
Occurrence 

Weeks To 
Resolve 

Maul tours 
To Resolve 












Figure 19. Risk area impact questionnaire. 















Activity: 

Scheduled Duration: 

Scheduled Monhours: 

No Problem Duration: 

No Problem Manhours: 

Expected Number of Problems 
Probability That a Problem is Identified: 


NO. 

Problem Description 

Probability of 
Occurrence 

Severity 

Weeks To 
Resolve 

Manhours 
To Resolve 

1. 



Optimistic 
Most Likely 
Pessimistic 



2. 



Optimistic 
Most Likely 
Pessim istic 



3. 



Optimistic 
Most Likely 
Pessimistic 



4. 



Optimistic 
Most Likely 
Pessimistic 



5. 



Optimistic 
Most Likely 
Pessimistic 



6. 



Optimistic 
Most Likely 
Pessimistic 



7. 



Optimistic 
Most Likely 
Pessimistic 




OPTIONAL 

UNKNOWN 

PROBLEM 


Very Optimistic 
Optimistic 
Most Likely 
Pessimistic 
Very Pessimistic 




Figure 20. Activity risk summary sheet. 



juoiussasse ysu qnpaqos/jsoo/ieoiuqoa} 


I JOJ UIBiSuip MOJJ 'll 3jn§lj 



I 3SVIU 




















Finally, using a program called ACTIVE to simulate the occurrence of the problems on 
each respective activity, a VERT input file would be created. VERT would then simulate the 
project using only its capability to simulate a cost/schedule network with and-all node logic. The 
results, as shown in Figure 21, would include cumulative distributions on time and cost, and criti- 
cality information on activities. 

VII. SUMMARY AND RECOMMENDATIONS 


Based on a thorough literature search, the range of quantitative methods for program risk 
assessment have been presented. As shown in Figure 22, twelve distinct alternatives were dis- 
cussed. All methods are based on the Bayesian view of probability; they differ in how subjective 
probability is collected (level of detail, assumptions, distribution types, etc.) and how these proba- 
bilities are combined into an overall assessment of uncertainty. After reading through the tech- 
niques, the reader is aware that although “a risk assessment” can be done in a matter of several 
days, the truly comprehensive risk methods treat technical, cost, and schedule risks in an integrated 
(network-based) fashion and require at least one month of up-front development. The management 
benefits of integrated, network-based methods are worth the expense and waiting-time for the initial 
model output. 

The recommendations for NASA-MSFC based on discussions with Program Planning Office 
personnel and the contents of this handbook are: 

1 . Commit to performing integrated cost/schedule risk assessment on each program prior to 
releasing RFPs for the phase in question. Quick risk assessments are, of course, 
appropriate in certain circumstances. 

2. Require contractors to perform quantitative risk assessments as part of their proposal 
preparation effort. Require that these assessments be submitted as part of the technical 
volume or as a separate volume, or back-up document. Be explicit that meaningless 
LOW-MEDIUM-HIGH risk ratings are not acceptable, and that integrated methods are 
preferred. Require risk analysis be part of the system’s analysis/PM process after contract 
award. 

3. Select (or hire) a full-time risk analyst to be stationed in Program Planning with the 
following responsibilities: 

• Perform risk analyses on all PD studies, with early involvement with PM and study 
team. 

• Write risk analysis requirements for all RFPs. 

• Develop and document databases, questionnaires, and methods. 
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BASIS OF TECHNIQUES 

TIME TO 
IMPLEMENT 

SUBJECTIVE FACTORS AND/OR 
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SUBJECTIVE PROBABILITY & 
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1 • 2 DAYS 
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Figure 22. Summary of risk assessment methodology alternatives. 




• Plan evolution of tools, either in-house development or outside acquisition. 

• Train project control personnel to perform risk analyses. 

• Interface with other centers. 

• Consult with PMs, chief engineers on use of risk analysis on their projects. 

4. Commit to investing the time and money to build a state-of-the-art capability in program 
risk analysis at NASA-MSFC. 

• Give selected analyst one year to build background, learn to use tools you have 
(ARTEMIS, SLAM, and SAM), and to review 'available methods and computer pack- 
ages. 

• Consider purchase of network simulation package designed for risk analysis 

- RISNET 
^ - TRIAD-VERT 

5. Inform technical personnel in PD, and lab personnel supporting PD, about what risk 
analysis is and how they may be involved. Perhaps include some training in basic statis- 
tical concepts (classical and Bayesian) and generally encourage team-work, cooperation 
in generation of risk information. 

6. As experience is gained, consider expanding this handbook to include: 

- risk identification methods 

- risk management methods 

- lessons learned 

- case histories 

7. Consider expanding from one risk analyst to a risk and decision analysis group. 
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APPENDIX A 


PROGRAM SOURCES OF UNCERTAINTY 


The following is a generic list of areas of uncertainty likely to be encountered on new 
programs. The DoD acquisition directive or primary specification applicable to each area is con 
tained in parentheses beside each uncertainty. 

Standardization and Interoperability (4120.3, 2010.6, 5000.37) 

Quality Control (4155.1) 

Manpower Numbers, Grades, Speciality, Skills (5000.39) 

Support Cost (5000.39) 

Training Facilities (5000.39) 

Simulator Designs (5000.39) 

Crew Station Requirements (5000.39) 

Reliability and Maintainability (5000.40) 

Transportability (3224.1) 

System Safety (5000.36) 

Physical Security (3224.3) 

Chemical Survivability 

Nuclear Survivability 

Test Hardware Availability (5000.3) 

Test Hardware Applicability (5000.3) 

Availability of Materials 

Production Capability (Surge - 4005.11) 

Affordability 

Computer Resources (5000.29) 

Interface Requirements (5010.19) 

Data Communication (5010.19) 

Software Development (5010.19) 

Software Standardization (5010.19) 

Software Documentation (5010.19) 

Software Testing (5010.19) 

Software Update (5010.19) 

Data Management (4120.18) 

Metric Units (4120.18) 

Frequency Bands Allocation ( ) 

Energy Usage (4140.43) 



Environmental Impact (6050.1) 
Socioeconomic Programs (1 100. 1 1) 

C 2 Systems 

Interface Changes 
C 3 Approach 

“Core” Functional Requirements 
Warranties of Equipment 
Survivability 

Redundancy and Jam Proofing 
Service Operational Capability 
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APPENDIX B 


CANDIDATE RISK AREAS 


The following is a checklist of risk elements that may reflect program uncertainties. 
PERFORMANCE RISK VARIABLES 


State-of-the-Art Advance 


Risks involving technology development beyond present capabilities. Also including risks 

due to: 


Complexity/difficulty in meeting requirement 

Percent proven technology 

Experience in the field needed 

Lack of work on similar programs 

Special resources needed 

Operating environment 

Required theoretical analysis 

Degree difference from existing technology 

Technical Risk Sources 


Physical Properties 

Dynamics 

Stress 

Thermal 

Mass 

Power 

Vibration 

Material Properties 

Chemical 

Pneumatics 

Hydraulics 

Atmospheric 

Ionization 
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Radiation Properties (emission and reception) 

EMI 

Nuclear 

Optical 

Sound 

Magnetic 

Material Availability Risks 

Sufficient quantity when needed? 

Sufficient quality? 

Alternate vendors needed? 
Subcontractor/vendor stable? 

Testing/Modeling Risks 

Test facilities valid? 

Modeling techniques valid? 
Interpolation/extrapolation required? 

Integration/Interface Risks 

Adaptability 

Compatibility 

Controllability 

Deployability 

Detectability 

Design tolerances 

Interoperability (e.g., service branches, allies) 

Quality Control 

Safety 

Security 

Specifications 

Standards 

Survivability (chemical, nuclear hardness) 

Transportability 

Units 

Vulnerability 
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Program Personnel 


Sufficient skill levels available? 
Available when needed? 

Clearance adequate? 

Travel requirements understood? 

Software Design Risks 

Code estimates reasonable? 
Language version defined? 
Functional requirements complete? 
Configuration control defined? 

Test procedures complete? 

Critical modules identified? 
Hardware constraints defined? 

Safety Risks 

Testing hazardous to personnel? 
Operation hazardous to personnel? 

Security Risks 

Security of system 
Security of operating personnel 
Security of support personnel 
Detectability of communications 

Critical Failure Modes 

Reliability 
Maintainability 
Redundancy 
Fault Detection 
Fault Correction 

Energy/Environmental Risks 

Energy use high? 

Environmental impact high? 



Schedule Risk Variables 


Sensitivity to technical risk 
Sensitivity to cost risk 
Dependence on prior or concurrent results 
Availability of materials 
Availability of personnel 
Availability of test facilities 
Turn around time 
Communication delays/errors 
Uncontrollable events 
Strikes 

Weather, etc. 

Availability of production facilities 
Change in requirements 
Test failures 

Cost Risk Varibles 

Sensitivity to technical risk 
Sensitivity to supportability risk 
Sensitivity to schedule risk 

Macro-Economic conditions (e.g., inflation, decreased demand) 
Political/social climate (e.g., funding changes) 

Regulatory changes (e.g., new laws, regulationss) 
Uncontrollable events 
Weather, etc. 

Strikes 
Overhead rates 
G&A rates 

Supportability Risk Variables 

Manpower (customer/user) 

Numbers 

Rates 

Specialities 
Skill levels 
Training 
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Facilities 

Maintenance 
S imul ators/T rainers 
Crew stations 

R&M 

Transportability 
System safety 
Quantity control 

Production capability 

Spares 

Lead times 

Availability of materials 
Surge production capability 
Environmental impact 
Data management 
Formats 
Access 
Maintenance 

Transfer of responsibilities 
Interoperability between services/allies 
Alternate sources 
Configuration management 
Requirements changes 



APPENDIX C 


TIME-COST TRADE-OFF ANALYSIS 


Although not a risk assessment method, the critical path acceleration method is useful in es- 
tablishing the relationship between schedule and resources (e.g., engineering manhours) in any 
network. Such trade-off relationships may be used directly by the PM (or chief design engineer), or 
may be used as an input to a risk analysis model. 

The utility and limitations of the critical path method (CPM) in conjunction with project 
planning are well-known. Experience in technology assessment and conceptual design projects at 
Lockheed-Georgia showed that a major difficulty in planning was that start and end dates for a 
specific technology development varied from project to project. Such variation is due to a combina- 
tion of factors: (1) need dates are driven by the need date of the total system; (2) need dates are 
interrelated with the availability dates of other technologies; (3) different projects require different 
levels of technology advancement to meet requirements. In this environment, it is obvious that one 
type of useful schedule information a risk analyst can provide is: (1) how much a development 
schedule could be accelerated (technology limits); (2) which activities should be accelerated, and in 
what order; (3) what would be the cost of each increment of time-savings. 

The method for conducting such an analysis is known as “time-cost crashing: in OR litera- 
ture and textbooks. Even though the size of the networks expected to be analyzed was not large 
(10 to 20 activities), the frequency of conducting such analyses and the occasional “short-fuse” 
application led us to develop an automated program. The computer code is called CPM/CRASH 
and its input— output requirements are illustrated in Figure 23. The program logic is based on an 
algorithm described in Tufekci (1982), “A Flow-Preserving Algorithm for the Time/Cost Trade-off 
Problem.” 

CPM/CRASF1 works as follows. Inputs to the model are: (1) a network description of en- 
abling activities; (2) a linear time-cost trade-off curve for each activity. CPM/CRASH systematical- 
ly accelerates the development project, “crashing” the activity or activities on the current critical 
path which gives the maximum time reduction per dollar. Crashing stops when a critical path is en- 
countered in which all critical activities are at their maximum accelerated state. Zeleny (1982) has 
pointed out that the project time-cost trade-off curve is an “efficiency boundary” in the terminology 
of multicriteria decision making. Thus, the curve provided to the manager based on CPM/CRASH 
is a trade-off curve between two conflicting objectives, minimize time and minimize cost. R&D 
managers recognize the value of such information, being accustomed to trade-off curves for 
performance-related variables. 

The CPM/CRASH model depicted in Figure 23 was implemented in FORTRAN 77 on a 
Univac 1100-series mainframe at Lockheed-Georgia. The model can evaluate a CPM network with 
up to 149 activities and 99 nodes. The only labeling requirement on the nodes is that each 
activity’s tail node must be numbered less than the number assigned its head node. The only 
analytic assumption, and on$ that must be emphasized, is that time to complete an activity is a 
linear function of resources expended, within an interval running from “normal gost” to an extreme 
“crashed cost.” Implied in this assumption is that expenditures of resources above the crashed cost 
have no effect on time to complete and hence would not be considered as rational. 
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Figure 23. Process to determine a project time-cost trade-off curve. 
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